train_siqa_456_1760637829

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the siqa dataset. It achieves the following results on the evaluation set:

• Loss: 0.1750
• Num Input Tokens Seen: 60272064

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.03
• train_batch_size: 4
• eval_batch_size: 4
• seed: 456
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.5502	1.0	7518	0.5502	3015336
0.5555	2.0	15036	0.5497	6029736
0.561	3.0	22554	0.5538	9044064
0.5476	4.0	30072	0.5499	12056056
0.5613	5.0	37590	0.5505	15070152
0.5553	6.0	45108	0.5474	18083976
0.5437	7.0	52626	0.5493	21097056
0.5191	8.0	60144	0.5443	24109664
0.5795	9.0	67662	0.5431	27122784
0.5133	10.0	75180	0.4792	30139392
0.2126	11.0	82698	0.2088	33151800
0.1314	12.0	90216	0.1795	36165976
0.0211	13.0	97734	0.1756	39180248
0.1962	14.0	105252	0.1750	42193928
0.1675	15.0	112770	0.1752	45207272
0.1324	16.0	120288	0.1772	48219232
0.0547	17.0	127806	0.1810	51231624
0.0837	18.0	135324	0.1834	54245832
0.1029	19.0	142842	0.1833	57258952
0.046	20.0	150360	0.1834	60272064

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4