train_winogrande_42_1760637610

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:

• Loss: 0.2271
• Num Input Tokens Seen: 34142816

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.2324	2.0	16160	0.2316	3414464
0.2827	4.0	32320	0.2127	6826752
0.1961	6.0	48480	0.0789	10242576
0.0006	8.0	64640	0.0774	13657920
0.0721	10.0	80800	0.0987	17072880
0.0835	12.0	96960	0.1223	20487056
0.0	14.0	113120	0.1525	23899136
0.0	16.0	129280	0.1989	27313344
0.0	18.0	145440	0.2236	30728688
0.0	20.0	161600	0.2271	34142816

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4