Finished gpt2 2-12 grades: evens pretraining

README.md

@@ -12,12 +12,12 @@ model-index:
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/uds/Graded%20text%20simplification%20training/runs/fqk06uhr)
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/uds/Graded%20text%20simplification%20training/runs/0wk7divh)
 # text-simplification
 
-This model is a fine-tuned version of [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) on an unknown dataset.
+This model is a fine-tuned version of [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.5755
+- Loss: 0.5713
 
 ## Model description
 
@@ -42,17 +42,17 @@ The following hyperparameters were used during training:
 - seed: 42
 - optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: linear
-- num_epochs: 10
+- num_epochs: 5
 
 ### Training results
 
-| Training Loss | Epoch | Step | Validation Loss |
-|:-------------:|:-----:|:----:|:---------------:|
-| 0.6211        | 1.0   | 898  | 0.5805          |
-| 0.6157        | 2.0   | 1796 | 0.5763          |
-| 0.6128        | 3.0   | 2694 | 0.5742          |
-| 0.6102        | 4.0   | 3592 | 0.5738          |
-| 0.6073        | 5.0   | 4490 | 0.5755          |
+| Training Loss | Epoch | Step  | Validation Loss |
+|:-------------:|:-----:|:-----:|:---------------:|
+| 0.884         | 1.0   | 6044  | 0.6113          |
+| 0.6392        | 2.0   | 12088 | 0.5825          |
+| 0.6168        | 3.0   | 18132 | 0.5764          |
+| 0.6077        | 4.0   | 24176 | 0.5732          |
+| 0.6038        | 5.0   | 30220 | 0.5713          |
 
 
 ### Framework versions

gpt2-2-12-evens/adapter_config.json

@@ -3,6 +3,8 @@
   "auto_mapping": null,
   "base_model_name_or_path": "openai-community/gpt2",
   "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
   "fan_in_fan_out": false,
   "inference_mode": true,
   "init_lora_weights": true,
@@ -11,6 +13,7 @@
   "layers_to_transform": null,
   "loftq_config": {},
   "lora_alpha": 32,
+  "lora_bias": false,
   "lora_dropout": 0.01,
   "megatron_config": null,
   "megatron_core": "megatron.core",
@@ -20,9 +23,9 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "c_proj",
     "lm_head",
     "c_attn",
+    "c_proj",
     "c_fc"
   ],
   "task_type": "CAUSAL_LM",

gpt2-2-12-evens/adapter_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4b31e8fcd9c3000cfe7def36e0c3bd86d4a47c0a03656b9f085bf75401ba1275
-size 83581271
+oid sha256:cc4b139ea2f475df1eb0209deb63b9c8a6cf89bfe7a4b18d26718ae034ee28f4
+size 160776023

gpt2-grade-3-finetuned/adapter_config.json

@@ -23,10 +23,10 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "lm_head",
-    "c_proj",
+    "c_attn",
     "c_fc",
-    "c_attn"
+    "lm_head",
+    "c_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

gpt2-grade-3-finetuned/adapter_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a7eae108f5b1d9a72aa5cf696a5fd0b6e0654ccb7b638b85ad96cc374244c6eb
+oid sha256:0f3aa3e03bd1accebd9e8e9458423a78b09f5e7aa1a77d74ec7452e3fd2972f9
 size 160776023

gpt2-grade-5-finetuned/adapter_config.json

@@ -23,10 +23,10 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "lm_head",
-    "c_proj",
+    "c_attn",
     "c_fc",
-    "c_attn"
+    "lm_head",
+    "c_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

gpt2-grade-5-finetuned/adapter_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8d43880788644a7f5a455e4c7f67b6f77363678065630f467b34e076065961bc
+oid sha256:7ca0e1c4334a463fa63c8f116f298e2e46287a70bb485bba6168d4cf35703e27
 size 160776023

tokenizer.json

@@ -1,7 +1,19 @@
 {
   "version": "1.0",
-  "truncation": null,
-  "padding": null,
+  "truncation": {
+    "direction": "Right",
+    "max_length": 1024,
+    "strategy": "LongestFirst",
+    "stride": 0
+  },
+  "padding": {
+    "strategy": "BatchLongest",
+    "direction": "Left",
+    "pad_to_multiple_of": null,
+    "pad_id": 50256,
+    "pad_type_id": 0,
+    "pad_token": "<|endoftext|>"
+  },
   "added_tokens": [
     {
       "id": 50256,

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:233d27052f3274bf965374919fe126018adf5670e5b612f5f26c9ccbde9b8a09
-size 5496
+oid sha256:8d705ee6117b5836dd06dd9e001ea2f4f63982c2ab893e466f4d58988a5c6f90
+size 5368