train_winogrande_789_1760637958

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:

• Loss: 0.0520
• Num Input Tokens Seen: 38393344

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 789
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.0032	1.0	9090	0.0643	1919360
0.0616	2.0	18180	0.0520	3838064
0.0006	3.0	27270	0.0651	5755984
0.0001	4.0	36360	0.0752	7675760
0.1017	5.0	45450	0.0799	9596528
0.0005	6.0	54540	0.0652	11515248
0.0	7.0	63630	0.0784	13435888
0.0001	8.0	72720	0.0960	15356016
0.1771	9.0	81810	0.1135	17274448
0.0001	10.0	90900	0.1050	19194672
0.0	11.0	99990	0.0955	21115984
0.0	12.0	109080	0.1056	23036144
0.0	13.0	118170	0.1114	24955120
0.0	14.0	127260	0.1172	26874400
0.0	15.0	136350	0.1333	28793728
0.0	16.0	145440	0.1619	30713760
0.0	17.0	154530	0.1682	32634016
0.0	18.0	163620	0.1718	34554208
0.0	19.0	172710	0.1744	36474880
0.0	20.0	181800	0.1751	38393344

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4