train_cola_456_1760637818

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the cola dataset. It achieves the following results on the evaluation set:

• Loss: 0.1326
• Num Input Tokens Seen: 7334376

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 456
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.2387	1.0	1924	0.1821	366712
0.251	2.0	3848	0.1326	734016
0.0045	3.0	5772	0.1935	1100824
0.0773	4.0	7696	0.1617	1467248
0.0019	5.0	9620	0.2296	1834568
0.0005	6.0	11544	0.3203	2201464
0.0001	7.0	13468	0.3507	2568040
0.0001	8.0	15392	0.3453	2934360
0.0003	9.0	17316	0.3871	3301448
0.0004	10.0	19240	0.3988	3668312
0.0035	11.0	21164	0.3997	4034856
0.0	12.0	23088	0.6586	4401344
0.0	13.0	25012	0.4966	4767736
0.0	14.0	26936	0.5001	5134344
0.0	15.0	28860	0.4590	5501408
0.0	16.0	30784	0.4632	5867920
0.0	17.0	32708	0.5291	6234920
0.0	18.0	34632	0.5293	6601944
0.0	19.0	36556	0.5330	6968096
0.0	20.0	38480	0.5316	7334376

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4