modernbert-CGEdit-AAE_sv3_final

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.9271

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.1
• num_epochs: 40

Training results

Training Loss	Epoch	Step	Validation Loss
3.7290	1.0	152	0.9322
3.7188	2.0	304	0.9309
3.7007	3.0	456	0.9299
3.6970	4.0	608	0.9301
3.6006	5.0	760	0.9290
3.6544	6.0	912	0.9295
3.6527	7.0	1064	0.9290
3.6134	8.0	1216	0.9292
3.5959	9.0	1368	0.9276
3.5148	10.0	1520	0.9276
3.6227	11.0	1672	0.9280
3.5786	12.0	1824	0.9275
3.6104	13.0	1976	0.9279
3.5895	14.0	2128	0.9276
3.5172	15.0	2280	0.9273
3.6369	16.0	2432	0.9275
3.6047	17.0	2584	0.9274
3.5937	18.0	2736	0.9271
3.5756	19.0	2888	0.9273
3.5059	20.0	3040	0.9270
3.5772	21.0	3192	0.9272
3.5817	22.0	3344	0.9271
3.5915	23.0	3496	0.9271
3.5628	24.0	3648	0.9270
3.4554	25.0	3800	0.9271

Framework versions

• Transformers 5.0.0
• Pytorch 2.5.1+cu121
• Tokenizers 0.22.1