README.md

---
library_name: transformers
license: mit
language:
- en
base_model:
- microsoft/phi-2
---

# Model Card for ShAIkespear/Phi-2_DPO_M3_Base_Alt

A **LoRA-finetuned** and **Direct Preference Optimization (DPO)**–aligned variant of **microsoft/phi-2**, specialized for **multiple-choice question answering (MCQA)** with an emphasis on **STEM and general knowledge** domains.
This model represents the *alternative base configuration* of the final **M3 (balanced-then-DPO)** training pipeline from the *ShAIkespear* project. It preserves full precision for highest fidelity and further fine-tuning, without 8-bit quantization.

---

## Model Details

* **Developed by:** ShAIkespear team
* **Shared by:** ShAIkespear team
* **Model type:** Causal LM (Phi-2) with LoRA adapters; DPO-aligned
* **Languages:** English
* **License:** MIT
* **Finetuned from:** microsoft/phi-2

### Model Sources

* **Repository:** [2.8B-Phi-2-LLM-QA](https://github.com/EricSaikali/2.8B-Phi-2-LLM-QA)
* **Report:** *“ShAIkespear – How to replace TAs: A comprehensive study on letting LLMs answer your questions”*

---

## Uses

### Direct Use

* MCQA and educational Q&A (MMLU, OpenBookQA, ScienceQA).
* Alignment research — comparison between DPO training setups (Base vs. Quantized).
* As a **high-fidelity reference checkpoint** for quantized and downstream variants.

### Out-of-Scope Use

* High-stakes or safety-critical applications (medical, legal, policy).
* Generative tasks outside multiple-choice reasoning.
* Misuse in automated exam solving or confidential data leakage.

---

## Bias, Risks, and Limitations

* **Domain bias:** Stronger on factual MCQA, weaker on advanced reasoning tasks.
* **Answer drift:** May occasionally produce verbose or follow-up answers without explicit formatting.
* **Data source risks:** EPFL-derived preferences may encode narrow style biases.

### Recommendations

* Maintain the structured prompt format:

  ```
  ### Question ...
  ### Explanation ...
  ### Answer:
  ```
* Keep human supervision in any educational or grading use.
* Prefer this full-precision model for fine-tuning or evaluation; use quantized versions for deployment.

---

## How to Get Started

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ShAIkespear/Phi-2_DPO_M3_Base_Alt"

tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "### Question: Which element has the chemical symbol 'O'?\n### Explanation: The symbol 'O' represents this essential gas.\n### Answer:"
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=15)
print(tok.decode(out[0], skip_special_tokens=True))
```

---

## Training Details

### Training Data

* **SFT stage:** Balanced MCQA mix — MathQA, OpenBookQA, ScienceQA, TAL-SCQ5K, and EPFL question sets.
* **DPO stage:** Human preference pairs (EPFL exams + public feedback datasets like HelpSteer).
* **Schema:** Unified “### Question / ### Explanation / ### Answer” format.
* **Filtering:** ≤512 tokens, balanced sample caps (~20k per dataset).

### Training Procedure

* **Pipeline:** SFT → DPO (M3 configuration).
* **LoRA parameters:** rank = 16, α = 16, dropout = 0.05.
* **Batch sizes:** SFT = 4; DPO = 1.
* **Learning rates:** 1e-5 (public) / 1e-4 (EPFL).
* **Scheduler:** Cosine with warmup.
* **Frameworks:** Hugging Face Transformers + TRL + PEFT (LoRA).

---

## Evaluation Summary

* **Configuration:** *M3 Base (Alt)* is the unquantized reference model for the quantized 8-bit variant.
* **Performance:** Balanced dataset improves cross-domain consistency; DPO enhances answer formatting and style alignment.
* **Accuracy:** Similar to quantized model (~0.61 MMLU avg.), slightly higher on reasoning subtasks.
* **Use case:** For experimentation, evaluation, or further domain-specific fine-tuning.

---

## Technical Specifications

* **Architecture:** Phi-2 (~2.78B parameters), decoder-only transformer.
* **Objective:** SFT next-token prediction + DPO preference alignment.
* **Precision:** Full precision (fp16/bf16).
* **Software:** Hugging Face Transformers, TRL, PEFT.

---

## Glossary

* **MCQA:** Multiple-Choice Question Answering
* **SFT:** Supervised Finetuning
* **DPO:** Direct Preference Optimization
* **LoRA:** Low-Rank Adaptation
* **Alt (Alternative):** Internal naming for the alternate full-precision checkpoint variant of M3