Upload folder using huggingface_hub

README.md

@@ -0,0 +1,57 @@
+---
+language: en
+license: mit
+datasets:
+- glue/rte
+tags:
+- text-classification
+- glue
+- bert
+- recognizing textual entailment
+- assignment
+- mean-pooling
+metrics:
+- accuracy
+---
+
+# BERT + Mean Pooling + MLP for RTE (EEE 486/586 Assignment - Part 2)
+
+This model is a fine-tuned version of `bert-base-uncased` on the RTE (Recognizing Textual Entailment) task from the GLUE benchmark. It was developed as part of the EEE 486/586 Statistical Foundations of Natural Language Processing course assignment (Part 2).
+
+## Model Architecture
+
+This model explores an alternative to the standard `BertForSequenceClassification` architecture:
+
+- Uses the standard `bert-base-uncased` model to obtain token embeddings (`last_hidden_state`).
+- **Mean Pooling:** Instead of using the [CLS] token's pooler output, it calculates the mean of the `last_hidden_state` across all non-padding tokens (using the attention mask) to get a single sequence representation vector.
+- **MLP Classifier Head:** The mean-pooled representation is passed through dropout and then a multi-layer perceptron (MLP) head for classification. The MLP structure was determined by the hyperparameter search (`hidden_size_multiplier=4`).
+- The final layer outputs logits for the 2 classes (entailment/not\_entailment).
+
+**Note:** Because this uses a custom architecture (`BertMeanPoolClassifier`), it cannot be loaded directly using `AutoModelForSequenceClassification.from_pretrained()`. You need the model's class definition (provided in the assignment code/report) and then load the `state_dict` (`pytorch_model.bin`) into an instance of that class.
+
+## Performance
+
+The model was trained using hyperparameters found via Optuna. The final training run (5 epochs with early stopping based on validation accuracy) achieved the following:
+
+- **Best Validation Accuracy:** **0.6931** (achieved at Epoch 3)
+- Final Validation Accuracy (Epoch 5): 0.6823
+- Final Validation Loss (Epoch 5): 1.4258
+- Final Training Loss (Epoch 5): 0.0797
+
+The model showed strong fitting capabilities but exhibited signs of overfitting after epoch 3, as indicated by the rising validation loss. The best checkpoint based on accuracy was saved.
+
+## Best Hyperparameters (from Optuna)
+
+| Hyperparameter           | Value                 |
+|--------------------------|-----------------------|
+| Learning Rate            | 3.518e-05 |
+| Max Sequence Length      | 128           |
+| Dropout Rate (Classifier)| 0.4          |
+| Batch Size               | 16             |
+| Hidden Size Multiplier   | 4         |
+| Epochs (Optuna Best Trial) | 3             |
+
+## Intended Use & Limitations
+
+This model is intended for the RTE task as part of the specific course assignment. Due to its custom architecture, direct loading via `AutoModelForSequenceClassification` is not supported.
+```

config.json

@@ -0,0 +1,40 @@
+{
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "custom_params": {
+    "batch_size": 16,
+    "dropout_rate": 0.4,
+    "hidden_size_multiplier": 4,
+    "learning_rate": 3.518086017884403e-05,
+    "max_length": 128,
+    "num_epochs": 3
+  },
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "not_entailment",
+    "1": "entailment"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "entailment": 1,
+    "not_entailment": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.50.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53092141b28e86ab154d3d4ac24a2144224c9b31ae9c2a7a0e94971b2f0bcf59
+size 456910918

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}