train_winogrande_123_1760637732

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:

• Loss: 0.0593
• Num Input Tokens Seen: 38394016

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.1212	1.0	9090	0.1526	1918144
0.0631	2.0	18180	0.0915	3838192
0.1906	3.0	27270	0.0742	5757648
0.0219	4.0	36360	0.0665	7676976
0.0536	5.0	45450	0.0630	9596496
0.1068	6.0	54540	0.0596	11516256
0.178	7.0	63630	0.0603	13435600
0.0075	8.0	72720	0.0593	15356752
0.0673	9.0	81810	0.0628	17276752
0.0109	10.0	90900	0.0609	19196064
0.0035	11.0	99990	0.0643	21115472
0.0088	12.0	109080	0.0673	23035440
0.0006	13.0	118170	0.0677	24955600
0.0063	14.0	127260	0.0682	26875344
0.0012	15.0	136350	0.0733	28795600
0.0784	16.0	145440	0.0737	30715008
0.0106	17.0	154530	0.0772	32634912
0.004	18.0	163620	0.0775	34554080
0.0084	19.0	172710	0.0772	36472448
0.0011	20.0	181800	0.0760	38394016

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4