QSolver_Encoder_V2

QSolver_Encoder_V2 is an encoder-based language model fine-tuned for 5-choice scientific multiple-choice question answering. Built on top of microsoft/deberta-v3-large, the model utilizes partial layer freezing, dynamic 3D sequence collating, 8-bit AdamW optimization, and custom Adversarial Weight Perturbation (AWP) to prevent overfitting and improve generalization across distribution shifts.

Evaluation is performed using 5-Fold Stratified Cross-Validation monitored with Mean Average Precision at 3 (MAP@3).

---

Model Details

• Model Name: QSolver_Encoder_V2
• Repository ID: dahaludba/QSolver_Encoder_V2
• Base Model: microsoft/deberta-v3-large
• Architecture: AutoModelForMultipleChoice
• Maximum Sequence Length: 512 tokens
• Optimization Precision: 32-bit Floating Point (FP32)
• Optimizer: 8-bit AdamW (optim="adamw_8bit" via bitsandbytes)
• Total Training Duration: 13 hours 5 minutes
• License: MIT License

---

Training Strategy & Features

1. Partial Encoder Layer Freezing

To preserve low-level semantic representation and prevent catastrophic forgetting, the bottom 12 encoder layers (deberta.encoder.layer.0 through 11) out of 24 layers were completely frozen. Only the top 12 layers (12 through 23) and the multiple-choice classification head were fine-tuned.

2. Adversarial Weight Perturbation (AWP)

To increase regularization and combat overfitting on small datasets, training uses a custom AWPTrainer.

• Activation: Starts after completing epoch 1.0.
• Target Layers: Active word_embeddings parameters.
• Step Size ($\alpha$): 1e-3
• Epsilon Bound ($\epsilon$): 1e-2 (limits weight perturbation magnitude to within 1% of original parameter weights).

During each backward pass, weight embeddings are perturbed in the direction of maximum loss increase, gradients are computed, and original clean weights are restored prior to executing the optimizer step.

3. Dynamic 3D Sequence Collating

Question-option pairs are tokenized into 5 choice paths per sample. A CustomDataCollator dynamically pads sequences within each batch and reshapes tensors into 3D shapes (batch_size, num_choices, max_seq_len) for compute-efficient batch passes through AutoModelForMultipleChoice.

---

Hyperparameters

Hyperparameter	Value
Peak Learning Rate	8e-6
LR Scheduler	Cosine Decay
Warmup Steps	30
Optimizer	adamw_8bit (bitsandbytes)
Weight Decay	0.01
Per Device Train Batch Size	1
Per Device Eval Batch Size	1
Gradient Accumulation Steps	16 (Effective Batch Size = 16)
Training Epochs	4 per fold
Floating Point Precision	Full FP32 (fp16=False)
Gradient Checkpointing	Enabled (use_reentrant=False, use_cache=False)
Frozen Layers	Encoder layers 0 to 11
Metric Monitored	MAP@3 (eval_map@3)
Seed	42

---

Experiment Tracking & Cross-Validation Results

Total training across all 5 folds completed in 13 hours and 5 minutes. Full run logs, learning rate curves, and loss trajectories are available via Weights & Biases:

• Fold 1: W&B Run chh1vxqe
• Fold 2: W&B Run eu537lgm
• Fold 3: W&B Run t3gjczof
• Fold 4: W&B Run qkukzynk
• Fold 5: W&B Run 92ksg28n

---

Evaluation Metric

Performance is measured using Mean Average Precision at 3 (MAP@3):

$$\text{MAP@3}=\frac{1}{U}\sum _{i=1}^U\sum _{k=1}^{\min (P,3)}P(k)\times \text{rel}(k)\backslash text\{MAP@3\}\; =\; \backslash frac\{1\}\{U\}\; \backslash sum\_\{i=1\}^\{U\}\; \backslash sum\_\{k=1\}^\{\backslash min(P,\; 3)\}\; P(k)\; \backslash times\; \backslash text\{rel\}(k)$$

Where $P(k)$ is the precision at rank $k$, and $\text{rel}(k)$ indicates whether the option at rank $k$ is the ground-truth label.

---

How to Load and Run Inference

import torch
import numpy as np
import itertools
from transformers import AutoTokenizer, AutoModelForMultipleChoice

REPO_ID = "dahaludba/QSolver_Encoder_V2"
SUBFOLDER = "fold_1"

# Load Tokenizer & Model
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-large")
model = AutoModelForMultipleChoice.from_pretrained(REPO_ID, subfolder=SUBFOLDER)
model.eval()

# Sample Question and Options
question = "Which organelle is responsible for cellular respiration in eukaryotic cells?"
options = [
    "Lysosome",
    "Mitochondria",
    "Chloroplast",
    "Golgi Apparatus",
    "Peroxisome"
]

# Pair Question with Options
first_sentences = [question] * 5
second_sentences = options

# Tokenize paired sequences
inputs = tokenizer(
    first_sentences,
    second_sentences,
    truncation=True,
    max_length=512,
    padding=True,
    return_tensors="pt"
)

# Reshape inputs to 3D tensors (batch_size=1, num_choices=5, seq_len)
batch_inputs = {
    k: v.unsqueeze(0) for k, v in inputs.items()
}

with torch.no_grad():
    outputs = model(**batch_inputs)
    logits = outputs.logits.cpu().numpy()[0]

# Rank top choices
option_letters = ["A", "B", "C", "D", "E"]
top3_indices = np.argsort(-logits)[:3]
top3_predictions = [f"{option_letters[idx]}: {options[idx]}" for idx in top3_indices]

print("Top 3 Predictions:", top3_predictions)

---

License

This model and all repository contents are distributed under the MIT License.