End of training

README.md

@@ -1,6 +1,4 @@
 ---
-license: mit
-base_model: gpt2
 tags:
 - generated_from_trainer
 model-index:
@@ -13,9 +11,9 @@ should probably proofread and complete it, then remove this comment. -->
 
 # gpt2-wikitext2
 
-This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on an unknown dataset.
+This model was trained from scratch on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 5.1612
+- Loss: 0.1674
 
 ## Model description
 
@@ -38,8 +36,8 @@ The following hyperparameters were used during training:
 - train_batch_size: 32
 - eval_batch_size: 32
 - seed: 42
-- gradient_accumulation_steps: 2
-- total_train_batch_size: 64
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 128
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 100
@@ -48,13 +46,13 @@ The following hyperparameters were used during training:
 
 ### Training results
 
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 6.5373        | 0.9982 | 281  | 5.8409          |
-| 5.3438        | 2.0    | 563  | 5.4300          |
-| 4.8327        | 2.9982 | 844  | 5.2438          |
-| 4.4037        | 4.0    | 1126 | 5.1676          |
-| 4.1464        | 4.9911 | 1405 | 5.1612          |
+| Training Loss | Epoch | Step  | Validation Loss |
+|:-------------:|:-----:|:-----:|:---------------:|
+| 1.4222        | 1.0   | 2916  | 0.7671          |
+| 0.7233        | 2.0   | 5832  | 0.5135          |
+| 0.4245        | 3.0   | 8748  | 0.2865          |
+| 0.2039        | 4.0   | 11664 | 0.1831          |
+| 0.1082        | 5.0   | 14580 | 0.1674          |
 
 
 ### Framework versions

config.json

@@ -1,15 +1,17 @@
 {
-  "_name_or_path": "gpt2",
+  "_name_or_path": "wiki-text3",
   "activation_function": "gelu_new",
   "architectures": [
     "GPT2LMHeadModel"
   ],
   "attn_pdrop": 0.1,
   "bos_token_id": 0,
+  "do_sample": true,
   "embd_pdrop": 0.1,
   "eos_token_id": 0,
   "initializer_range": 0.02,
   "layer_norm_epsilon": 1e-05,
+  "max_length": 50,
   "model_type": "gpt2",
   "n_ctx": 128,
   "n_embd": 768,

generation_config.json

@@ -1,6 +1,9 @@
 {
   "_from_model_config": true,
   "bos_token_id": 0,
+  "do_sample": true,
   "eos_token_id": 0,
+  "max_length": 50,
+  "pad_token_id": 25000,
   "transformers_version": "4.41.2"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:927e3a1ebf36cd6413958be9a064a0f5488f89ce029c1958adab5c587bb5cefb
+oid sha256:5bba1402cd738060a965730133693ab300ceae7aeaec3d34066ce3d02297009b
 size 165013384

runs/Jun04_03-41-42_ip-10-192-11-48/events.out.tfevents.1717472506.ip-10-192-11-48.2329.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04132fb1b2f75f605c9ca88f5e88442026960f243fb785c71c51632e998a80b0
+size 5538

runs/Jun04_03-53-21_ip-10-192-11-48/events.out.tfevents.1717473202.ip-10-192-11-48.2329.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a628edd42bea13fa8b97922b38e9c386281015e673ca053c97f70a28158a119e
+size 5063

runs/Jun04_03-55-20_ip-10-192-11-48/events.out.tfevents.1717473322.ip-10-192-11-48.2329.3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0c3370b1099780ad9d15a0a3e533a27437f199c24aef4d2f5302b9b1dc285bd
+size 5885

runs/Jun04_03-57-25_ip-10-192-11-48/events.out.tfevents.1717473446.ip-10-192-11-48.2329.4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c13d676009ec4c882832801c0c628c6eca7d28a6bc7eb978ac65754c2a73e15
+size 7868

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c6b315b710e89a17665fd04487a1a99ae900757a72a5395d9d61f6b958fa6620
+oid sha256:28e7a5aa26195abb3a45c60622411497e74291083da19dafb1aa43c25d0993d3
 size 5112