gpt2_u020_tiny-stories_1024_dpos

This model is a fine-tuned version of on the roneneldan/TinyStories dataset. It achieves the following results on the evaluation set:

• Loss: 1.2112
• Accuracy: 0.6789

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.9273	0.0523	1000	2.4701	0.4442
1.9839	0.1047	2000	1.8112	0.5669
1.7325	0.1570	3000	1.6165	0.6014
1.6124	0.2093	4000	1.5144	0.6200
1.5362	0.2617	5000	1.4518	0.6320
1.4867	0.3140	6000	1.4061	0.6405
1.4484	0.3663	7000	1.3687	0.6476
1.4143	0.4187	8000	1.3439	0.6522
1.3866	0.4710	9000	1.3193	0.6572
1.3679	0.5233	10000	1.2997	0.6608
1.3469	0.5756	11000	1.2814	0.6647
1.3313	0.6280	12000	1.2682	0.6672
1.3156	0.6803	13000	1.2560	0.6696
1.3073	0.7326	14000	1.2458	0.6717
1.2944	0.7850	15000	1.2347	0.6740
1.2873	0.8373	16000	1.2268	0.6756
1.2768	0.8896	17000	1.2200	0.6770
1.2735	0.9420	18000	1.2145	0.6782
1.2681	0.9943	19000	1.2112	0.6789

Framework versions

• Transformers 4.42.3
• Pytorch 2.2.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1