train_winogrande_456_1760637844

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:

• Loss: 0.0534
• Num Input Tokens Seen: 38395408

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 456
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.1507	1.0	9090	0.0772	1919808
0.0043	2.0	18180	0.0534	3839104
0.0014	3.0	27270	0.0625	5758016
0.0967	4.0	36360	0.0612	7678560
0.0001	5.0	45450	0.0544	9598912
0.0	6.0	54540	0.0930	11518656
0.0	7.0	63630	0.0872	13438320
0.0001	8.0	72720	0.0920	15358064
0.0001	9.0	81810	0.0821	17278064
0.1188	10.0	90900	0.0977	19196144
0.0	11.0	99990	0.1242	21117200
0.0	12.0	109080	0.1071	23037584
0.0	13.0	118170	0.1202	24956720
0.0	14.0	127260	0.1026	26875344
0.0	15.0	136350	0.1193	28793344
0.0	16.0	145440	0.1572	30713568
0.0	17.0	154530	0.1624	32635088
0.0	18.0	163620	0.1642	34555376
0.0	19.0	172710	0.1662	36474544
0.0	20.0	181800	0.1654	38395408

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4