README.md

---
license: mit
base_model: microsoft/deberta-v3-large
library_name: transformers
pipeline_tag: text-classification
tags:
- deberta
- deberta-v3
- multiple-choice
- question-answering
- awp
metrics:
- map@3
---

# QSolver_Encoder_V2

QSolver_Encoder_V2 is an encoder-based language model fine-tuned for 5-choice scientific multiple-choice question answering. Built on top of `microsoft/deberta-v3-large`, the model utilizes partial layer freezing, dynamic 3D sequence collating, 8-bit AdamW optimization, and custom **Adversarial Weight Perturbation (AWP)** to prevent overfitting and improve generalization across distribution shifts.

Evaluation is performed using 5-Fold Stratified Cross-Validation monitored with Mean Average Precision at 3 (MAP@3).

---

## Model Details

- **Model Name:** QSolver_Encoder_V2
- **Repository ID:** `dahaludba/QSolver_Encoder_V2`
- **Base Model:** `microsoft/deberta-v3-large`
- **Architecture:** `AutoModelForMultipleChoice`
- **Maximum Sequence Length:** 512 tokens
- **Optimization Precision:** 32-bit Floating Point (FP32)
- **Optimizer:** 8-bit AdamW (`optim="adamw_8bit"` via `bitsandbytes`)
- **Total Training Duration:** 13 hours 5 minutes
- **License:** MIT License

---

## Training Strategy & Features

### 1. Partial Encoder Layer Freezing
To preserve low-level semantic representation and prevent catastrophic forgetting, the bottom 12 encoder layers (`deberta.encoder.layer.0` through `11`) out of 24 layers were completely frozen. Only the top 12 layers (`12` through `23`) and the multiple-choice classification head were fine-tuned.

### 2. Adversarial Weight Perturbation (AWP)
To increase regularization and combat overfitting on small datasets, training uses a custom `AWPTrainer`. 
- **Activation:** Starts after completing epoch 1.0.
- **Target Layers:** Active `word_embeddings` parameters.
- **Step Size ($\alpha$):** `1e-3`
- **Epsilon Bound ($\epsilon$):** `1e-2` (limits weight perturbation magnitude to within 1% of original parameter weights).

During each backward pass, weight embeddings are perturbed in the direction of maximum loss increase, gradients are computed, and original clean weights are restored prior to executing the optimizer step.

### 3. Dynamic 3D Sequence Collating
Question-option pairs are tokenized into 5 choice paths per sample. A `CustomDataCollator` dynamically pads sequences within each batch and reshapes tensors into 3D shapes `(batch_size, num_choices, max_seq_len)` for compute-efficient batch passes through `AutoModelForMultipleChoice`.

---

## Hyperparameters

| Hyperparameter | Value |
| :--- | :--- |
| Peak Learning Rate | 8e-6 |
| LR Scheduler | Cosine Decay |
| Warmup Steps | 30 |
| Optimizer | `adamw_8bit` (bitsandbytes) |
| Weight Decay | 0.01 |
| Per Device Train Batch Size | 1 |
| Per Device Eval Batch Size | 1 |
| Gradient Accumulation Steps | 16 (Effective Batch Size = 16) |
| Training Epochs | 4 per fold |
| Floating Point Precision | Full FP32 (`fp16=False`) |
| Gradient Checkpointing | Enabled (`use_reentrant=False`, `use_cache=False`) |
| Frozen Layers | Encoder layers 0 to 11 |
| Metric Monitored | MAP@3 (`eval_map@3`) |
| Seed | 42 |

---

## Experiment Tracking & Cross-Validation Results

Total training across all 5 folds completed in **13 hours and 5 minutes**. Full run logs, learning rate curves, and loss trajectories are available via Weights & Biases:

- **Fold 1:** [W&B Run chh1vxqe](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/chh1vxqe?nw=nwuser24f2002963)
- **Fold 2:** [W&B Run eu537lgm](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/eu537lgm?nw=nwuser24f2002963)
- **Fold 3:** [W&B Run t3gjczof](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/t3gjczof?nw=nwuser24f2002963)
- **Fold 4:** [W&B Run qkukzynk](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/qkukzynk?nw=nwuser24f2002963)
- **Fold 5:** [W&B Run 92ksg28n](https://wandb.ai/24f2002963-dl-genai-project/24f2002963-t22026/runs/92ksg28n?nw=nwuser24f2002963)

---

## Evaluation Metric

Performance is measured using Mean Average Precision at 3 (MAP@3):

$$\text{MAP@3} = \frac{1}{U} \sum_{i=1}^{U} \sum_{k=1}^{\min(P, 3)} P(k) \times \text{rel}(k)$$

Where $P(k)$ is the precision at rank $k$, and $\text{rel}(k)$ indicates whether the option at rank $k$ is the ground-truth label.

---

## How to Load and Run Inference

```python
import torch
import numpy as np
import itertools
from transformers import AutoTokenizer, AutoModelForMultipleChoice

REPO_ID = "dahaludba/QSolver_Encoder_V2"
SUBFOLDER = "fold_1"

# Load Tokenizer & Model
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-large")
model = AutoModelForMultipleChoice.from_pretrained(REPO_ID, subfolder=SUBFOLDER)
model.eval()

# Sample Question and Options
question = "Which organelle is responsible for cellular respiration in eukaryotic cells?"
options = [
    "Lysosome",
    "Mitochondria",
    "Chloroplast",
    "Golgi Apparatus",
    "Peroxisome"
]

# Pair Question with Options
first_sentences = [question] * 5
second_sentences = options

# Tokenize paired sequences
inputs = tokenizer(
    first_sentences,
    second_sentences,
    truncation=True,
    max_length=512,
    padding=True,
    return_tensors="pt"
)

# Reshape inputs to 3D tensors (batch_size=1, num_choices=5, seq_len)
batch_inputs = {
    k: v.unsqueeze(0) for k, v in inputs.items()
}

with torch.no_grad():
    outputs = model(**batch_inputs)
    logits = outputs.logits.cpu().numpy()[0]

# Rank top choices
option_letters = ["A", "B", "C", "D", "E"]
top3_indices = np.argsort(-logits)[:3]
top3_predictions = [f"{option_letters[idx]}: {options[idx]}" for idx in top3_indices]

print("Top 3 Predictions:", top3_predictions)
```

---

## License

This model and all repository contents are distributed under the **MIT License**.