ninagroot/GPT2-705Mtest

README.md

@@ -0,0 +1,98 @@
+---
+tags:
+- generated_from_trainer
+model-index:
+- name: GPT2-705M
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# GPT2-705M
+
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 4.1692
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.00025
+- train_batch_size: 16
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 8
+- total_train_batch_size: 128
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 50
+- num_epochs: 40
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 6.9278        | 1.0   | 3    | 6.8982          |
+| 6.2619        | 2.0   | 6    | 6.3736          |
+| 5.5361        | 3.0   | 9    | 6.0513          |
+| 5.3332        | 4.0   | 12   | 5.3785          |
+| 5.5741        | 5.0   | 15   | 5.3255          |
+| 4.6819        | 6.0   | 18   | 4.9017          |
+| 4.2971        | 7.0   | 21   | 4.5924          |
+| 4.1379        | 8.0   | 24   | 4.3890          |
+| 4.014         | 9.0   | 27   | 4.1882          |
+| 3.8136        | 10.0  | 30   | 3.9852          |
+| 3.6263        | 11.0  | 33   | 3.8839          |
+| 3.4951        | 12.0  | 36   | 3.8463          |
+| 3.1734        | 13.0  | 39   | 3.7403          |
+| 3.3178        | 14.0  | 42   | 3.6179          |
+| 2.9869        | 15.0  | 45   | 3.5469          |
+| 2.8614        | 16.0  | 48   | 3.4918          |
+| 2.7445        | 17.0  | 51   | 3.4338          |
+| 2.6014        | 18.0  | 54   | 3.4071          |
+| 2.5147        | 19.0  | 57   | 3.4175          |
+| 2.5795        | 20.0  | 60   | 3.3969          |
+| 2.38          | 21.0  | 63   | 3.4091          |
+| 2.1579        | 22.0  | 66   | 3.4791          |
+| 2.202         | 23.0  | 69   | 3.5244          |
+| 2.0225        | 24.0  | 72   | 3.5956          |
+| 1.9019        | 25.0  | 75   | 3.6241          |
+| 1.8023        | 26.0  | 78   | 3.6849          |
+| 1.7232        | 27.0  | 81   | 3.7259          |
+| 1.6271        | 28.0  | 84   | 3.8172          |
+| 1.4719        | 29.0  | 87   | 3.8680          |
+| 1.3646        | 30.0  | 90   | 3.9362          |
+| 1.2769        | 31.0  | 93   | 3.9921          |
+| 1.2041        | 32.0  | 96   | 4.0741          |
+| 1.0735        | 33.0  | 99   | 4.0879          |
+| 1.042         | 34.0  | 102  | 4.1016          |
+| 0.9854        | 35.0  | 105  | 4.1211          |
+| 0.9374        | 36.0  | 108  | 4.1667          |
+| 0.8552        | 37.0  | 111  | 4.1515          |
+| 0.8455        | 38.0  | 114  | 4.1612          |
+| 0.8147        | 39.0  | 117  | 4.1692          |
+| 0.7733        | 40.0  | 120  | 4.1692          |
+
+
+### Framework versions
+
+- Transformers 4.39.1
+- Pytorch 2.1.2+cu121
+- Datasets 2.16.1
+- Tokenizers 0.15.0

config.json

@@ -0,0 +1,32 @@
+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.0,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 1536,
+  "n_head": 8,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 256,
+  "pad_token_id": 0,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.0,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.39.1",
+  "use_cache": true,
+  "vocab_size": 4388
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "pad_token_id": 0,
+  "transformers_version": "4.39.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b43c9a47230fe97124a40fb61d15f2b6f4c9a806433103a83e28c12d92a5196
+size 2748401440

runs/Apr18_20-17-44_gcn59.local.snellius.surf.nl/events.out.tfevents.1713464276.gcn59.local.snellius.surf.nl.992823.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0bae0680c3dc222401c4fd40bfb7e81ad124c6294bf805beb0a57fd29cc89c8
+size 40505

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebf14cdba58dbfbada67ce9a4a8ab11e55fe5977cad0cb6058aee1b75371f6ec
+size 4984