train_winogrande_456_1760637841

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:

• Loss: 0.2975
• Num Input Tokens Seen: 34139376

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 456
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.2421	2.0	16160	0.2330	3412976
0.2431	4.0	32320	0.2274	6826432
0.0578	6.0	48480	0.1222	10241072
0.0731	8.0	64640	0.0792	13657088
0.0003	10.0	80800	0.1145	17069152
0.0007	12.0	96960	0.1136	20481536
0.0001	14.0	113120	0.1711	23896640
0.0	16.0	129280	0.2107	27311488
0.0	18.0	145440	0.2798	30725088
0.0	20.0	161600	0.2975	34139376

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4