pretrain_spdl_

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5261

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1024
• eval_batch_size: 1024
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 156250
• num_epochs: 25
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.4617	0.3774	500	1.7630
0.3667	0.7547	1000	1.3955
0.3332	1.1321	1500	1.2666
0.3104	1.5094	2000	1.1712
0.2941	1.8868	2500	1.0907
0.2766	2.2642	3000	1.0196
0.2606	2.6415	3500	0.9639
0.2505	3.0189	4000	0.9049
0.2359	3.3962	4500	0.8557
0.2251	3.7736	5000	0.8215
0.2169	4.1509	5500	0.7879
0.2126	4.5283	6000	0.7684
0.2049	4.9057	6500	0.7461
0.1997	5.2830	7000	0.7286
0.1962	5.6604	7500	0.7119
0.19	6.0377	8000	0.7050
0.1878	6.4151	8500	0.6906
0.1825	6.7925	9000	0.6831
0.1816	7.1698	9500	0.6669
0.1748	7.5472	10000	0.6604
0.1741	7.9245	10500	0.6518
0.1707	8.3019	11000	0.6429
0.1727	8.6792	11500	0.6378
0.1688	9.0566	12000	0.6326
0.166	9.4340	12500	0.6273
0.1652	9.8113	13000	0.6163
0.1642	10.1887	13500	0.6110
0.162	10.5660	14000	0.6068
0.1634	10.9434	14500	0.6044
0.1601	11.3208	15000	0.5986
0.1599	11.6981	15500	0.5942
0.1584	12.0755	16000	0.5909
0.1562	12.4528	16500	0.5863
0.1544	12.8302	17000	0.5832
0.1553	13.2075	17500	0.5801
0.1518	13.5849	18000	0.5778
0.1535	13.9623	18500	0.5737
0.1525	14.3396	19000	0.5727
0.1512	14.7170	19500	0.5710
0.1511	15.0943	20000	0.5675
0.1503	15.4717	20500	0.5671
0.1504	15.8491	21000	0.5640
0.1499	16.2264	21500	0.5618
0.1497	16.6038	22000	0.5584
0.1464	16.9811	22500	0.5552
0.148	17.3585	23000	0.5557
0.1465	17.7358	23500	0.5523
0.1465	18.1132	24000	0.5513
0.1447	18.4906	24500	0.5487
0.1452	18.8679	25000	0.5479
0.1447	19.2453	25500	0.5452
0.1431	19.6226	26000	0.5439
0.1438	20.0	26500	0.5430
0.1437	20.3774	27000	0.5411
0.1428	20.7547	27500	0.5395
0.1434	21.1321	28000	0.5388
0.142	21.5094	28500	0.5362
0.1418	21.8868	29000	0.5347
0.1419	22.2642	29500	0.5345
0.1418	22.6415	30000	0.5321
0.1407	23.0189	30500	0.5312
0.141	23.3962	31000	0.5303
0.1392	23.7736	31500	0.5293
0.1384	24.1509	32000	0.5289
0.1372	24.5283	32500	0.5274
0.1392	24.9057	33000	0.5261

Framework versions

• Transformers 4.51.1
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1