train_winogrande_42_1760637616

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:

• Loss: 6.6528
• Num Input Tokens Seen: 38397712

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
7.3369	1.0	9090	7.1380	1918960
6.4914	2.0	18180	6.7209	3839712
6.5564	3.0	27270	6.6718	5759216
6.4719	4.0	36360	6.6662	7678944
6.5774	5.0	45450	6.6543	9598112
6.5414	6.0	54540	6.6572	11518608
6.4149	7.0	63630	6.6562	13438816
6.9295	8.0	72720	6.6574	15359200
6.5697	9.0	81810	6.6670	17280320
6.6917	10.0	90900	6.6608	19200384
6.8079	11.0	99990	6.6644	21120032
6.6893	12.0	109080	6.6528	23039856
6.4786	13.0	118170	6.6610	24959536
6.8271	14.0	127260	6.6610	26879696
6.6678	15.0	136350	6.6610	28798160
6.5871	16.0	145440	6.6610	30718896
6.7357	17.0	154530	6.6610	32638160
6.5054	18.0	163620	6.6610	34558000
6.6281	19.0	172710	6.6610	36477680
6.6692	20.0	181800	6.6610	38397712

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4