train_codealpacapy_123_1762496345

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the codealpacapy dataset. It achieves the following results on the evaluation set:

• Loss: 0.6313
• Num Input Tokens Seen: 22139032

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.6065	2.0	3392	0.5241	2216560
0.4471	4.0	6784	0.5017	4419816
0.5516	6.0	10176	0.5020	6632800
0.3557	8.0	13568	0.5099	8848168
0.4427	10.0	16960	0.5294	11063280
0.4077	12.0	20352	0.5488	13281912
0.2964	14.0	23744	0.5795	15500632
0.3286	16.0	27136	0.6075	17712104
0.3527	18.0	30528	0.6274	19926608
0.2493	20.0	33920	0.6313	22139032

Framework versions

• PEFT 0.15.2
• Transformers 4.51.3
• Pytorch 2.8.0+cu128
• Datasets 3.6.0
• Tokenizers 0.21.1