train_winogrande_456_1760637843

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:

• Loss: 7.7509
• Num Input Tokens Seen: 38395408

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 456
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.2314	1.0	9090	0.2314	1919808
0.2293	2.0	18180	0.2313	3839104
0.0548	3.0	27270	0.0630	5758016
0.0102	4.0	36360	0.0553	7678560
0.0165	5.0	45450	0.0496	9598912
0.1573	6.0	54540	0.0538	11518656
0.0236	7.0	63630	0.0517	13438320
0.0067	8.0	72720	0.0513	15358064
0.0525	9.0	81810	0.0533	17278064
0.0969	10.0	90900	0.0536	19196144
0.0123	11.0	99990	0.0590	21117200
0.0374	12.0	109080	0.0586	23037584
0.0008	13.0	118170	0.0694	24956720
0.0025	14.0	127260	0.0751	26875344
0.0004	15.0	136350	0.0813	28793344
0.0005	16.0	145440	0.0870	30713568
0.0029	17.0	154530	0.0918	32635088
0.0003	18.0	163620	0.0975	34555376
0.0009	19.0	172710	0.1017	36474544
0.0002	20.0	181800	0.1023	38395408

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4