metadata
library_name: peft
license: llama3
base_model: meta-llama/Meta-Llama-3-8B-Instruct
tags:
- base_model:adapter:meta-llama/Meta-Llama-3-8B-Instruct
- llama-factory
- transformers
pipeline_tag: text-generation
model-index:
- name: train_winogrande_42_1760637613
results: []
train_winogrande_42_1760637613
This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the winogrande dataset. It achieves the following results on the evaluation set:
- Loss: 9.0314
- Num Input Tokens Seen: 38397712
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 20
Training results
| Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen |
|---|---|---|---|---|
| 0.2316 | 1.0 | 9090 | 0.2342 | 1918960 |
| 0.2206 | 2.0 | 18180 | 0.2526 | 3839712 |
| 0.2561 | 3.0 | 27270 | 0.2372 | 5759216 |
| 0.2161 | 4.0 | 36360 | 0.2406 | 7678944 |
| 0.2338 | 5.0 | 45450 | 0.2366 | 9598112 |
| 0.233 | 6.0 | 54540 | 0.2320 | 11518608 |
| 0.2265 | 7.0 | 63630 | 0.2352 | 13438816 |
| 0.2307 | 8.0 | 72720 | 0.2338 | 15359200 |
| 0.2298 | 9.0 | 81810 | 0.2323 | 17280320 |
| 0.2307 | 10.0 | 90900 | 0.2318 | 19200384 |
| 0.2329 | 11.0 | 99990 | 0.2317 | 21120032 |
| 0.2288 | 12.0 | 109080 | 0.2313 | 23039856 |
| 0.2295 | 13.0 | 118170 | 0.2316 | 24959536 |
| 0.2371 | 14.0 | 127260 | 0.2315 | 26879696 |
| 0.2282 | 15.0 | 136350 | 0.2325 | 28798160 |
| 0.2296 | 16.0 | 145440 | 0.2315 | 30718896 |
| 0.2362 | 17.0 | 154530 | 0.2315 | 32638160 |
| 0.2265 | 18.0 | 163620 | 0.2315 | 34558000 |
| 0.2327 | 19.0 | 172710 | 0.2316 | 36477680 |
| 0.2316 | 20.0 | 181800 | 0.2315 | 38397712 |
Framework versions
- PEFT 0.17.1
- Transformers 4.51.3
- Pytorch 2.9.0+cu128
- Datasets 4.0.0
- Tokenizers 0.21.4