metadata
tags:
- generated_from_trainer
datasets:
- roneneldan/TinyStories
metrics:
- accuracy
model-index:
- name: gpt2_m050_tiny-stories_1024
results:
- task:
name: Causal Language Modeling
type: text-generation
dataset:
name: roneneldan/TinyStories
type: roneneldan/TinyStories
metrics:
- name: Accuracy
type: accuracy
value: 0.6794915189952896
gpt2_m050_tiny-stories_1024
This model is a fine-tuned version of on the roneneldan/TinyStories dataset. It achieves the following results on the evaluation set:
- Loss: 1.2035
- Accuracy: 0.6795
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 2.9083 | 0.0525 | 1000 | 2.4374 | 0.4486 |
| 1.9644 | 0.1049 | 2000 | 1.7837 | 0.5703 |
| 1.7149 | 0.1574 | 3000 | 1.5991 | 0.6031 |
| 1.5979 | 0.2099 | 4000 | 1.5038 | 0.6204 |
| 1.5248 | 0.2623 | 5000 | 1.4431 | 0.6322 |
| 1.4723 | 0.3148 | 6000 | 1.3973 | 0.6411 |
| 1.4339 | 0.3672 | 7000 | 1.3621 | 0.6475 |
| 1.406 | 0.4197 | 8000 | 1.3340 | 0.6530 |
| 1.3764 | 0.4722 | 9000 | 1.3089 | 0.6579 |
| 1.3561 | 0.5246 | 10000 | 1.2903 | 0.6618 |
| 1.3357 | 0.5771 | 11000 | 1.2739 | 0.6649 |
| 1.3213 | 0.6296 | 12000 | 1.2586 | 0.6680 |
| 1.3081 | 0.6820 | 13000 | 1.2466 | 0.6704 |
| 1.2962 | 0.7345 | 14000 | 1.2362 | 0.6726 |
| 1.2867 | 0.7869 | 15000 | 1.2277 | 0.6744 |
| 1.2755 | 0.8394 | 16000 | 1.2186 | 0.6762 |
| 1.2709 | 0.8919 | 17000 | 1.2117 | 0.6776 |
| 1.2611 | 0.9443 | 18000 | 1.2070 | 0.6787 |
| 1.2628 | 0.9968 | 19000 | 1.2035 | 0.6795 |
Framework versions
- Transformers 4.42.3
- Pytorch 2.2.2+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1