train_gsm8k_42_1760637594

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the gsm8k dataset. It achieves the following results on the evaluation set:

• Loss: 5.3493
• Num Input Tokens Seen: 34797032

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.5392	1.0	1682	0.5075	1738904
0.5614	2.0	3364	0.4831	3481872
0.5138	3.0	5046	0.4693	5222160
0.4085	4.0	6728	0.4629	6964040
0.4671	5.0	8410	0.4581	8703920
0.3373	6.0	10092	0.4520	10444208
0.4354	7.0	11774	0.4488	12184024
0.4815	8.0	13456	0.4464	13925976
0.4923	9.0	15138	0.4468	15667472
0.4457	10.0	16820	0.4447	17407120
0.4029	11.0	18502	0.4435	19145240
0.3926	12.0	20184	0.4444	20882704
0.414	13.0	21866	0.4441	22623936
0.3982	14.0	23548	0.4473	24361736
0.3849	15.0	25230	0.4467	26106632
0.334	16.0	26912	0.4483	27845256
0.3922	17.0	28594	0.4499	29579704
0.4437	18.0	30276	0.4503	31322600
0.3183	19.0	31958	0.4517	33057576
0.4014	20.0	33640	0.4521	34797032

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4