README.md

metadata

library_name: peft
license: llama3
base_model: meta-llama/Meta-Llama-3-8B-Instruct
tags:
  - llama-factory
  - lora
  - generated_from_trainer
model-index:
  - name: train_codealpacapy_1754507520
    results: []

train_codealpacapy_1754507520

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the codealpacapy dataset. It achieves the following results on the evaluation set:

• Loss: 0.4662
• Num Input Tokens Seen: 12472912

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10.0

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.4542	0.5	954	0.4947	616992
0.581	1.0	1908	0.4810	1248304
0.4509	1.5	2862	0.4745	1877040
0.4172	2.0	3816	0.4662	2497016
0.5189	2.5	4770	0.4698	3129368
0.3859	3.0	5724	0.4691	3742552
0.3728	3.5	6678	0.4796	4361944
0.3182	4.0	7632	0.4808	4985200
0.3908	4.5	8586	0.4979	5611760
0.3861	5.0	9540	0.5026	6233920
0.3046	5.5	10494	0.5369	6849184
0.3087	6.0	11448	0.5363	7478504
0.1964	6.5	12402	0.5860	8083560
0.2524	7.0	13356	0.5758	8722744
0.1868	7.5	14310	0.6237	9345976
0.5188	8.0	15264	0.6234	9977520
0.1183	8.5	16218	0.6655	10604656
0.2188	9.0	17172	0.6605	11225416
0.152	9.5	18126	0.6764	11845704
0.277	10.0	19080	0.6786	12472912

Framework versions

• PEFT 0.15.2
• Transformers 4.51.3
• Pytorch 2.8.0+cu128
• Datasets 3.6.0
• Tokenizers 0.21.1