train_cola_123_1760637704

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the cola dataset. It achieves the following results on the evaluation set:

• Loss: 0.1535
• Num Input Tokens Seen: 7337920

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.1742	1.0	1924	0.2045	367320
0.2706	2.0	3848	0.1535	734600
0.3173	3.0	5772	0.1616	1101216
0.0864	4.0	7696	0.1727	1468552
0.0004	5.0	9620	0.2625	1834816
0.0038	6.0	11544	0.2210	2201584
0.0005	7.0	13468	0.2889	2568288
0.0004	8.0	15392	0.3569	2935056
0.0001	9.0	17316	0.3409	3301760
0.0001	10.0	19240	0.3469	3669168
0.0	11.0	21164	0.5200	4036096
0.0	12.0	23088	0.4918	4403128
0.0088	13.0	25012	0.3633	4769264
0.0	14.0	26936	0.4145	5136352
0.0	15.0	28860	0.4740	5503048
0.0	16.0	30784	0.5514	5869824
0.0	17.0	32708	0.5584	6236752
0.0	18.0	34632	0.5592	6603776
0.0	19.0	36556	0.5726	6970736
0.0	20.0	38480	0.5668	7337920

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4