Finished gpt2 2-12 grades: all pretraining

README.md

@@ -12,12 +12,12 @@ model-index:
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/uds/Graded%20text%20simplification%20training/runs/osneb4fp)
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/uds/Graded%20text%20simplification%20training/runs/z2eqo7kv)
 # text-simplification
 
 This model is a fine-tuned version of [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.6210
+- Loss: 0.5581
 
 ## Model description
 
@@ -40,21 +40,25 @@ The following hyperparameters were used during training:
 - train_batch_size: 16
 - eval_batch_size: 16
 - seed: 42
-- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: linear
-- num_epochs: 1
+- num_epochs: 5
 
 ### Training results
 
-| Training Loss | Epoch | Step | Validation Loss |
-|:-------------:|:-----:|:----:|:---------------:|
-| 0.8184        | 1.0   | 9864 | 0.6210          |
+| Training Loss | Epoch | Step  | Validation Loss |
+|:-------------:|:-----:|:-----:|:---------------:|
+| 0.7915        | 1.0   | 9864  | 0.5974          |
+| 0.6181        | 2.0   | 19728 | 0.5722          |
+| 0.6011        | 3.0   | 29592 | 0.5637          |
+| 0.5938        | 4.0   | 39456 | 0.5591          |
+| 0.5902        | 5.0   | 49320 | 0.5581          |
 
 
 ### Framework versions
 
 - PEFT 0.14.0
-- Transformers 4.48.2
-- Pytorch 2.2.0a0+81ea7a4
+- Transformers 4.48.3
+- Pytorch 2.6.0+cu124
 - Datasets 3.2.0
 - Tokenizers 0.21.0

gpt2-2-12-all/adapter_config.json

@@ -23,10 +23,10 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "c_fc",
     "c_attn",
+    "c_proj",
     "lm_head",
-    "c_proj"
+    "c_fc"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

gpt2-2-12-all/adapter_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d42e480566052d2adecb1f124b4638c99427d746481d04e36931c179e31cb8dc
+oid sha256:209849fbebd2fb9d4b2a23c7d7605c6c951a265e10bf9067c6fc2226ddbfe22c
 size 160776023

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:061c1d68b23812e84cc2947119f2a3e8cda1418e66b7c5b1854b83f1a5dc9877
+oid sha256:81d58a712d8d32d4e159d1bec24ead7fae6605c50be3cf4024eff6af00573442
 size 5368