train_winogrande_101112_1760638072

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:

• Loss: 6.6503
• Num Input Tokens Seen: 38366624

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 101112
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
7.0733	1.0	9090	7.1428	1917952
7.0367	2.0	18180	6.7191	3835840
6.4538	3.0	27270	6.6629	5753152
6.8846	4.0	36360	6.6679	7672000
6.6273	5.0	45450	6.6657	9590080
6.5984	6.0	54540	6.6512	11509088
6.5612	7.0	63630	6.6503	13427712
6.546	8.0	72720	6.6538	15346672
6.8749	9.0	81810	6.6599	17265344
6.6863	10.0	90900	6.6577	19184224
6.7185	11.0	99990	6.6619	21102912
6.5521	12.0	109080	6.6589	23021312
6.6488	13.0	118170	6.6554	24938688
6.6391	14.0	127260	6.6554	26857088
6.6688	15.0	136350	6.6554	28775840
6.5522	16.0	145440	6.6554	30693088
6.4774	17.0	154530	6.6554	32612480
6.9787	18.0	163620	6.6554	34530176
6.8303	19.0	172710	6.6554	36447600
6.7336	20.0	181800	6.6554	38366624

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4