Smart MCQ Solver: DeBERTa-v3-large, fp32

README.md

@@ -0,0 +1,81 @@
+---
+license: mit
+base_model: microsoft/deberta-v3-large
+tags: [multiple-choice, question-answering, deberta-v3, mcq]
+library_name: transformers
+pipeline_tag: multiple-choice
+---
+
+# Smart MCQ Solver - DeBERTa-v3-large
+
+Five-option multiple-choice QA over science and philosophy questions.
+Full fine-tune of `microsoft/deberta-v3-large` with `AutoModelForMultipleChoice`.
+
+## Results
+
+| Metric | Value |
+|---|---|
+| 3-fold grouped CV MAP@3 | **0.7567** |
+| Held-out MAP@3 (this artifact) | 0.7944 |
+| Held-out accuracy | 0.6912 |
+| Random MAP@3 baseline | 0.3667 |
+
+Cross-validation is `GroupKFold` grouped by normalised prompt, because the dataset
+contains roughly eight near-duplicate phrasings of every question; a random split
+would score memorisation rather than generalisation.
+
+## Loading
+
+**Always pass `dtype=torch.float32` explicitly.**
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForMultipleChoice
+
+tok = AutoTokenizer.from_pretrained("SriragData/smart-mcq-deberta-v3-large")
+model = AutoModelForMultipleChoice.from_pretrained(
+    "SriragData/smart-mcq-deberta-v3-large", dtype=torch.float32).eval()
+
+question = "What is the capital of France?"
+options  = ["Berlin", "Madrid", "Paris", "Rome", "Lisbon"]
+enc = tok([question]*5, options, truncation=True, max_length=256,
+          padding=True, return_tensors="pt")
+with torch.no_grad():
+    logits = model(input_ids=enc["input_ids"].unsqueeze(0),
+                   attention_mask=enc["attention_mask"].unsqueeze(0)).logits[0]
+print(options[int(logits.argmax())])
+```
+
+## Why the dtype matters
+
+The upstream `microsoft/deberta-v3-*` checkpoints store fp16 weights, and
+transformers 5.x honours the dtype recorded in the checkpoint. Loaded in fp16,
+DeBERTa's disentangled attention saturates, the attention softmax collapses to
+one-hot, and the model returns an identical score for every option - MAP@3 drops
+to roughly 0.36, the random baseline, with no error raised. This repository stores
+fp32 weights and records `float32` in `config.json`, but passing `dtype` explicitly
+costs nothing and removes the risk entirely.
+
+## Training
+
+| Setting | Value |
+|---|---|
+| Precision | full fp32 (no mixed precision) |
+| Batch size | 1, gradient accumulation 16 (effective 16) |
+| Epochs | {EPOCHS} |
+| Learning rate | 8e-6, cosine schedule, warmup ratio 0.1 |
+| Max length | 256 |
+| Gradient checkpointing | enabled |
+| Checkpointing | disabled (`save_strategy="no"`) |
+
+`save_strategy="no"` is deliberate: on some transformers versions the checkpoint
+save/reload cycle renames LayerNorm `gamma`/`beta` and silently resets every
+LayerNorm to its initial values after training completes.
+
+## Limitations
+
+- Trained on 2,000 rows covering 252 unique questions. Coverage is narrow.
+- The evaluation set overlaps the training set heavily, so headline scores reflect
+  that deployment condition. The leakage-free estimate for this project is 0.6817.
+- Closed-book only: no retrieval, no citation, no abstention. It will answer
+  confidently on questions it knows nothing about.

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "[MASK]": 128000
+}

config.json

@@ -0,0 +1,35 @@
+{
+  "_name_or_path": "/kaggle/working/models/deberta_large",
+  "architectures": [
+    "DebertaV2ForMultipleChoice"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-07,
+  "max_position_embeddings": 512,
+  "max_relative_positions": -1,
+  "model_type": "deberta-v2",
+  "norm_rel_ebd": "layer_norm",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 0,
+  "pooler_dropout": 0,
+  "pooler_hidden_act": "gelu",
+  "pooler_hidden_size": 1024,
+  "pos_att_type": [
+    "p2c",
+    "c2p"
+  ],
+  "position_biased_input": false,
+  "position_buckets": 256,
+  "relative_attention": true,
+  "share_att_key": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.44.2",
+  "type_vocab_size": 0,
+  "vocab_size": 128100
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80c976de162bee63283074a958ba05b8ca48a24f7f9469ac901d0fb17edc5fe6
+size 1740300340

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "[CLS]",
+  "cls_token": "[CLS]",
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
+size 2464616

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128000": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
+}