gpt2_u000_tiny-stories_1024_dpos

This model is a fine-tuned version of on the roneneldan/TinyStories dataset. It achieves the following results on the evaluation set:

• Loss: 1.2096
• Accuracy: 0.6786

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.9186	0.0524	1000	2.4481	0.4472
1.9778	0.1048	2000	1.7944	0.5688
1.7276	0.1572	3000	1.6169	0.6007
1.6066	0.2095	4000	1.5123	0.6198
1.533	0.2619	5000	1.4486	0.6316
1.4848	0.3143	6000	1.4045	0.6401
1.4405	0.3667	7000	1.3674	0.6470
1.4133	0.4191	8000	1.3405	0.6520
1.3857	0.4715	9000	1.3192	0.6562
1.365	0.5238	10000	1.2987	0.6602
1.3463	0.5762	11000	1.2799	0.6642
1.3293	0.6286	12000	1.2664	0.6667
1.3183	0.6810	13000	1.2539	0.6693
1.3013	0.7334	14000	1.2438	0.6713
1.295	0.7858	15000	1.2328	0.6737
1.2821	0.8381	16000	1.2249	0.6754
1.2768	0.8905	17000	1.2182	0.6767
1.2726	0.9429	18000	1.2129	0.6779
1.2667	0.9953	19000	1.2097	0.6786

Framework versions

• Transformers 4.42.3
• Pytorch 2.2.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1