train_siqa_456_1760637830

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the siqa dataset. It achieves the following results on the evaluation set:

• Loss: 2.1480
• Num Input Tokens Seen: 60272064

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 456
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.2159	1.0	7518	0.1946	3015336
0.1279	2.0	15036	0.1866	6029736
0.1681	3.0	22554	0.1812	9044064
0.0846	4.0	30072	0.1737	12056056
0.4453	5.0	37590	0.1755	15070152
0.1321	6.0	45108	0.1720	18083976
0.1205	7.0	52626	0.1782	21097056
0.2912	8.0	60144	0.1786	24109664
0.124	9.0	67662	0.1957	27122784
0.0814	10.0	75180	0.2063	30139392
0.0331	11.0	82698	0.2217	33151800
0.1298	12.0	90216	0.2503	36165976
0.0118	13.0	97734	0.2616	39180248
0.0609	14.0	105252	0.3086	42193928
0.0086	15.0	112770	0.3349	45207272
0.0021	16.0	120288	0.3971	48219232
0.0022	17.0	127806	0.4191	51231624
0.0028	18.0	135324	0.4447	54245832
0.0008	19.0	142842	0.4527	57258952
0.0009	20.0	150360	0.4602	60272064

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4