train_cola_123_1760637707

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the cola dataset. It achieves the following results on the evaluation set:

• Loss: 0.1514
• Num Input Tokens Seen: 7337920

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.2892	1.0	1924	0.2023	367320
0.193	2.0	3848	0.1679	734600
0.3037	3.0	5772	0.1656	1101216
0.2236	4.0	7696	0.1607	1468552
0.1154	5.0	9620	0.1677	1834816
0.0886	6.0	11544	0.1529	2201584
0.0708	7.0	13468	0.1565	2568288
0.1078	8.0	15392	0.1582	2935056
0.1769	9.0	17316	0.1546	3301760
0.2451	10.0	19240	0.1517	3669168
0.0484	11.0	21164	0.1526	4036096
0.2081	12.0	23088	0.1572	4403128
0.0958	13.0	25012	0.1514	4769264
0.1993	14.0	26936	0.1543	5136352
0.2033	15.0	28860	0.1539	5503048
0.0991	16.0	30784	0.1523	5869824
0.0881	17.0	32708	0.1536	6236752
0.2064	18.0	34632	0.1529	6603776
0.1527	19.0	36556	0.1545	6970736
0.0773	20.0	38480	0.1536	7337920

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4