README.md

---
library_name: transformers
license: mit
language:
- en
base_model:
- microsoft/phi-2
---

# Model Card for ShAIkespear/Phi-2_DPO_M3_Quantized

A **quantized (8-bit)**, **LoRA-finetuned** variant of **microsoft/phi-2** specialized for **multiple-choice question answering (MCQA)**, particularly in **STEM and general knowledge** domains.
This model represents the final **Direct Preference Optimization (DPO)** stage of the *ShAIkespear* project, fine-tuned on both public MCQA datasets and EPFL preference-annotated data, then quantized to 8-bit for efficient inference and deployment.

---

## Model Details

* **Developed by:** ShAIkespear team
* **Shared by:** ShAIkespear team
* **Model type:** Causal LM (Phi-2) with LoRA adapters; DPO-aligned and 8-bit quantized
* **Languages:** English
* **License:** MIT
* **Finetuned from:** microsoft/phi-2

### Model Sources

* **Repository:** [2.8B-Phi-2-LLM-QA](https://github.com/EricSaikali/2.8B-Phi-2-LLM-QA)
* **Report:** *“ShAIkespear – How to replace TAs: A comprehensive study on letting LLMs answer your questions”*

---

## Uses

### Direct Use

* Lightweight, low-memory MCQA reasoning for STEM and general knowledge domains.
* Educational tutoring or automated evaluation assistants following structured prompts.
* Deployment on GPUs with limited VRAM (8-bit quantization reduces memory from ~11 GB → ~3 GB).

### Out-of-Scope Use

* Critical decision-making (medical, legal, financial).
* Long-form reasoning or open-ended creative writing.
* Any application violating academic integrity or confidentiality of test materials.

---

## Bias, Risks, and Limitations

* **Quantization trade-off:** Slight loss in accuracy compared to full-precision base model.
* **STEM reasoning:** Difficult multi-step math/science questions may still yield near-random performance (~25 % accuracy).
* **Alignment drift:** DPO may slightly overfit stylistic preferences or verbosity.

### Recommendations

* Use structured prompts (`### Question → ### Explanation → ### Answer`) for best results.
* Include human oversight for evaluation or teaching uses.
* Avoid deployment where model-generated answers have direct consequences.

---

## How to Get Started

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_cfg = BitsAndBytesConfig(load_in_8bit=True)
tok = AutoTokenizer.from_pretrained("ShAIkespear/Phi-2_DPO_M3_Quantized", use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    "ShAIkespear/Phi-2_DPO_M3_Quantized", device_map="auto", quantization_config=bnb_cfg
)

prompt = "### Question: What planet is known as the Red Planet?\n### Explanation: Identify the planet with a reddish appearance.\n### Answer:"
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=15)
print(tok.decode(out[0], skip_special_tokens=True))
```

---

## Training Details

### Training Data

* **SFT stage:** Mixed MCQA sets — MathQA, OpenBookQA, ScienceQA, TAL-SCQ5K, and EPFL-curated questions.
* **DPO stage:** Human preference pairs (EPFL exams + HelpSteer-style pairs).
* **Preprocessing:** Filtered to ≤512 tokens, unified MCQA schema.
* **Split:** 50 % train, 25 % overfit test, 10 % comparison, 15 % quantization validation.

### Training Procedure

* **Pipeline:** SFT → DPO → 8-bit quantization.
* **LoRA:** rank = 16, α = 16, dropout = 0.05.
* **Batch size:** 4 (SFT), 1 (DPO).
* **Learning rates:** 1e-5 (public), 1e-4 (EPFL).
* **Scheduler:** Cosine with warmup.
* **Frameworks:** Hugging Face Transformers + TRL + PEFT + BitsAndBytes.

---

## Evaluation Summary

* **Configuration:** “Balanced-then-DPO” (M3) achieved best overall performance.
* **Accuracy:** ≈ 0.61 on MMLU (balanced set); STEM tasks lower (~0.25).
* **Memory:** Reduced to ~3 GB with minor quality loss.
* **Outcome:** Best trade-off between efficiency and alignment across ShAIkespear models.

---

## Technical Specifications

* **Architecture:** Phi-2 (2.78 B parameters), decoder-only transformer.
* **Objective:** SFT next-token prediction + DPO preference alignment.
* **Quantization:** Post-training 8-bit (BitsAndBytes).
* **Precision:** 8-bit integer with dynamic quantization layers.
* **Software:** Hugging Face Transformers, TRL, PEFT, BitsAndBytes.

---

## Glossary

* **MCQA:** Multiple-Choice Question Answering
* **SFT:** Supervised Finetuning
* **DPO:** Direct Preference Optimization
* **LoRA:** Low-Rank Adaptation for efficient fine-tuning
* **Quantization:** Reducing model precision for faster, memory-efficient inference