pretrained_spdl_05_17

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.6339

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1024
• eval_batch_size: 1024
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 156250
• num_epochs: 9
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.4552	0.3774	500	1.7358
0.3672	0.7547	1000	1.4004
0.3344	1.1321	1500	1.2714
0.3115	1.5094	2000	1.1743
0.2946	1.8868	2500	1.0900
0.2768	2.2642	3000	1.0181
0.2602	2.6415	3500	0.9613
0.2507	3.0189	4000	0.9021
0.2365	3.3962	4500	0.8570
0.2248	3.7736	5000	0.8209
0.2172	4.1509	5500	0.7872
0.212	4.5283	6000	0.7676
0.2045	4.9057	6500	0.7501
0.1998	5.2830	7000	0.7287
0.1959	5.6604	7500	0.7104
0.1894	6.0377	8000	0.7057
0.1877	6.4151	8500	0.6901
0.1826	6.7925	9000	0.6747
0.1813	7.1698	9500	0.6666
0.1746	7.5472	10000	0.6579
0.1747	7.9245	10500	0.6481
0.1713	8.3019	11000	0.6404
0.1726	8.6792	11500	0.6339

Framework versions

• Transformers 4.51.1
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1