SriragData/smart-mcq-deberta-v3-large

Smart MCQ Solver - DeBERTa-v3-large

Five-option multiple-choice QA over science and philosophy questions. Full fine-tune of microsoft/deberta-v3-large with AutoModelForMultipleChoice.

Results

Metric	Value
3-fold grouped CV MAP@3	0.7567
Held-out MAP@3 (this artifact)	0.7944
Held-out accuracy	0.6912
Random MAP@3 baseline	0.3667

Cross-validation is GroupKFold grouped by normalised prompt, because the dataset contains roughly eight near-duplicate phrasings of every question; a random split would score memorisation rather than generalisation.

Loading

Always pass dtype=torch.float32 explicitly.

import torch
from transformers import AutoTokenizer, AutoModelForMultipleChoice

tok = AutoTokenizer.from_pretrained("SriragData/smart-mcq-deberta-v3-large")
model = AutoModelForMultipleChoice.from_pretrained(
    "SriragData/smart-mcq-deberta-v3-large", dtype=torch.float32).eval()

question = "What is the capital of France?"
options  = ["Berlin", "Madrid", "Paris", "Rome", "Lisbon"]
enc = tok([question]*5, options, truncation=True, max_length=256,
          padding=True, return_tensors="pt")
with torch.no_grad():
    logits = model(input_ids=enc["input_ids"].unsqueeze(0),
                   attention_mask=enc["attention_mask"].unsqueeze(0)).logits[0]
print(options[int(logits.argmax())])

Why the dtype matters

The upstream microsoft/deberta-v3-* checkpoints store fp16 weights, and transformers 5.x honours the dtype recorded in the checkpoint. Loaded in fp16, DeBERTa's disentangled attention saturates, the attention softmax collapses to one-hot, and the model returns an identical score for every option - MAP@3 drops to roughly 0.36, the random baseline, with no error raised. This repository stores fp32 weights and records float32 in config.json, but passing dtype explicitly costs nothing and removes the risk entirely.

Training

Setting	Value
Precision	full fp32 (no mixed precision)
Batch size	1, gradient accumulation 16 (effective 16)
Epochs	{EPOCHS}
Learning rate	8e-6, cosine schedule, warmup ratio 0.1
Max length	256
Gradient checkpointing	enabled
Checkpointing	disabled (save_strategy="no")

save_strategy="no" is deliberate: on some transformers versions the checkpoint save/reload cycle renames LayerNorm gamma/beta and silently resets every LayerNorm to its initial values after training completes.

Limitations

• Trained on 2,000 rows covering 252 unique questions. Coverage is narrow.
• The evaluation set overlaps the training set heavily, so headline scores reflect that deployment condition. The leakage-free estimate for this project is 0.6817.
• Closed-book only: no retrieval, no citation, no abstention. It will answer confidently on questions it knows nothing about.