End of training

README.md

@@ -1,51 +1,47 @@
----
-datasets:
-- HuggingFaceFW/fineweb
-language:
-- en
-library_name: transformers
-license: unlicense
-metrics:
-- perplexity
-pipeline_tag: text-generation
-tags:
-- text-generation-inference
-- casual-lm
-- question-answering
-- text-generation
----
-# Judge-GPT2
-
-Judge-GPT2 is a custom GPT-2 model designed for text generation tasks.
-
-## Model Information
-
-- **Name**: Judge-GPT2
-- **Version**: 1.0
-- **Description**: Judge-GPT2 is a variant of the GPT-2 model with custom configurations. It has been pretrained and finetuned for specific text generation tasks.
-
-## Model Details
-
-- **Architecture**: GPT-2
-- **Pre-trained weights**: [Wonder-Griffin/Judge-GPT2](https://huggingface.co/Wonder-Griffin/Judge-GPT2)
-- **Training data**: Fine-tuned on HuggingFaceFW/fineweb dataset
-
-## Model Performance
-
-- **Task**: Text generation
-- **Metrics**: Accuracy (not applicable), Perplexity (25.3)
-
-## Usage
-
-Here is an example of how to use the model for text generation:
-
-```python
-from transformers import AutoTokenizer, GPT2LMHeadModel
-
-tokenizer = AutoTokenizer.from_pretrained("Wonder-Griffin/Judge-GPT2")
-model = GPT2LMHeadModel.from_pretrained("Wonder-Griffin/Judge-GPT2")
-
-prompt = "Once upon a time"
-inputs = tokenizer(prompt, return_tensors="pt")
-outputs = model.generate(inputs['input_ids'], max_length=50, pad_token_id=tokenizer.eos_token_id)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+---
+base_model: Wonder-Griffin/JudgeLLM2
+tags:
+- generated_from_trainer
+model-index:
+- name: Judge-GPT2
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# Judge-GPT2
+
+This model is a fine-tuned version of [Wonder-Griffin/JudgeLLM2](https://huggingface.co/Wonder-Griffin/JudgeLLM2) on an unknown dataset.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 3.0
+
+### Framework versions
+
+- Transformers 4.43.3
+- Pytorch 2.4.0+cu124
+- Datasets 2.20.0
+- Tokenizers 0.19.1

config.json

@@ -1,15 +1,21 @@
 {
+  "_name_": "Judge-GPT2",
+  "_name_or_path": "Wonder-Griffin/JudgeLLM2",
+  "activation_function": "gelu_new",
   "architectures": [
-    "JudgeModel"
+    "GPT2LMHeadModel"
   ],
+  "attn_pdrop": 0.1,
   "batch_size": 32,
   "bias": true,
   "block_size": 512,
+  "bos_token_id": 50256,
   "dim_feedforward": 3072,
   "dropout": 0.1,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
   "ff_expansion_factor": 4,
   "hidden_act": "gelu",
-  "hidden_size": 768,
   "id2label": {
     "0": "LABEL_0",
     "1": "LABEL_1",
@@ -17,6 +23,8 @@
     "3": "LABEL_3",
     "4": "LABEL_4"
   },
+  "inference_mode": true,
+  "initializer_range": 0.02,
   "label2id": {
     "LABEL_0": 0,
     "LABEL_1": 1,
@@ -25,19 +33,70 @@
     "LABEL_4": 4
   },
   "label_smoothing": 0.1,
+  "layer_norm_epsilon": 1e-05,
   "learning_rate": 0.0003,
   "log_interval": 100,
   "max_grad_norm": 1.0,
-  "max_position_embeddings": 512,
   "model_type": "gpt2",
   "n_embd": 768,
   "n_head": 12,
+  "n_inner": null,
   "n_layer": 12,
+  "n_positions": 512,
   "output_dir": "C:/Users/wonde/output",
+  "pretrained_weights": "Wonder-Griffin/Judge-GPT2",
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_heads": {
+    "classifier_head": {
+      "params": {
+        "num_labels": 5
+      },
+      "type": "JudgeClassifier"
+    },
+    "lm_head": {
+      "params": {
+        "vocab_size": 50257
+      },
+      "type": "JudgeCasualLMHead"
+    },
+    "qa_head": {
+      "params": {
+        "num_labels": 2
+      },
+      "type": "JudgeWithQA"
+    }
+  },
+  "task_specific_params": {
+    "question-answering": {
+      "max_answer_length": 100
+    },
+    "sequence-classification": {
+      "eval_steps": 500
+    },
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 100
+    }
+  },
+  "tokenizer": {
+    "params": {
+      "vocab_size": 50257
+    },
+    "type": "AutoTokenizer"
+  },
   "torch_dtype": "float32",
   "total_steps": 10000,
   "transformers_version": "4.43.3",
-  "vocab_size": 50257,
+  "use_cache": true,
+  "vocab_size": 30522,
   "warmup_steps": 1000,
   "weight_decay": 0.01
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9401a9bdb241de29b6e8586a80d1899f25d3cdf9f52c200160c8b4aa22b36dce
-size 963697256
+oid sha256:6226a0b17d9cd2d41487623616804e90a3f3cfd31369dc6f676b1e7f4b47fa86
+size 435575424

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d66cf00c5d39fe4a0b6a597cc25e6b74ee22f8f2790b715f0d83753841f413a
+size 5176