metadata
library_name: peft
license: llama3
base_model: meta-llama/Meta-Llama-3-8B-Instruct
tags:
- base_model:adapter:meta-llama/Meta-Llama-3-8B-Instruct
- llama-factory
- transformers
pipeline_tag: text-generation
model-index:
- name: train_gsm8k_123_1760637708
results: []
train_gsm8k_123_1760637708
This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the gsm8k dataset. It achieves the following results on the evaluation set:
- Loss: 0.5359
- Num Input Tokens Seen: 30837792
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 123
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 20
Training results
| Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen |
|---|---|---|---|---|
| 0.4825 | 2.0 | 2990 | 0.5423 | 3073904 |
| 0.4576 | 4.0 | 5980 | 0.5077 | 6160576 |
| 0.5076 | 6.0 | 8970 | 0.4979 | 9246832 |
| 0.3616 | 8.0 | 11960 | 0.4952 | 12330800 |
| 0.4245 | 10.0 | 14950 | 0.4962 | 15414800 |
| 0.3441 | 12.0 | 17940 | 0.5092 | 18492000 |
| 0.3093 | 14.0 | 20930 | 0.5210 | 21579648 |
| 0.3145 | 16.0 | 23920 | 0.5284 | 24669056 |
| 0.3427 | 18.0 | 26910 | 0.5340 | 27752896 |
| 0.3472 | 20.0 | 29900 | 0.5359 | 30837792 |
Framework versions
- PEFT 0.17.1
- Transformers 4.51.3
- Pytorch 2.9.0+cu128
- Datasets 4.0.0
- Tokenizers 0.21.4