train_siqa_456_1760637833

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the siqa dataset. It achieves the following results on the evaluation set:

• Loss: 0.1825
• Num Input Tokens Seen: 60272064

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 456
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.259	1.0	7518	0.2497	3015336
0.2084	2.0	15036	0.2178	6029736
0.1573	3.0	22554	0.2038	9044064
0.1917	4.0	30072	0.1937	12056056
0.3725	5.0	37590	0.1889	15070152
0.1175	6.0	45108	0.1868	18083976
0.174	7.0	52626	0.1870	21097056
0.2971	8.0	60144	0.1825	24109664
0.2085	9.0	67662	0.1857	27122784
0.2811	10.0	75180	0.1843	30139392
0.2479	11.0	82698	0.1841	33151800
0.1492	12.0	90216	0.1853	36165976
0.0224	13.0	97734	0.1844	39180248
0.1808	14.0	105252	0.1845	42193928
0.1821	15.0	112770	0.1853	45207272
0.1169	16.0	120288	0.1855	48219232
0.0528	17.0	127806	0.1866	51231624
0.1685	18.0	135324	0.1867	54245832
0.1108	19.0	142842	0.1864	57258952
0.0491	20.0	150360	0.1866	60272064

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4