README.md

---
base_model: Qwen/Qwen3-1.7B
library_name: peft
license: apache-2.0
language:
  - en
pipeline_tag: text-generation
tags:
  - qlora
  - dpo
  - lora
  - education
  - synthetic-persona
---

# Believably-Wrong Student: Qwen3-1.7B (SFT + DPO LoRA)

A LoRA fine-tune of **Qwen/Qwen3-1.7B** that role-plays a realistic *struggling student*: given a
multiple-choice question (correct answer withheld), it reasons coherently, commits a recognizable
misconception, and selects a **wrong** option that follows from its own logic, reliably and in
character. This is `qwen3-wrong-dpo` (SFT then DPO).

**Thesis:** a well-prompted frontier model can't do this reliably (see Evaluation); the behavior comes
from the training data, not model scale. Full evaluation: `results/RESULTS.md`.

## Intended use & scope

- **In scope:** reasoning-style MCQs (quant / science-reasoning / logic) where a wrong answer can
  follow from a coherent misconception. Research/education use: generating realistic distractor
  reasoning, studying misconceptions, tutoring-system stress data.
- **Out of scope:** factual QA / assistance (it is *designed to be wrong*), pure-recall trivia, any
  setting where correctness matters. Not a general assistant.

## How to use

Install: `pip install transformers peft torch`. (If `peft` raises an "incompatible torchao" error, run
`pip uninstall -y torchao`; this model uses plain LoRA and does not need it.)

Load the base model plus this adapter:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "cd1313/qwen3-wrong-dpo")
```

The behavior depends on the **exact** training system prompt, with Qwen3 thinking mode OFF, the question
rendered with lettered options and the correct answer withheld, and a final `Answer: X` line. A
different prompt will mostly produce correct answers.

```python
SYSTEM_PROMPT = (
    "You are simulating one specific, realistic student who is capable but has "
    "genuine gaps in understanding. You will be given a multiple-choice question. "
    "Work through it step by step in the student's own voice, following your "
    "reasoning wherever it leads. Your reasoning contains one or more genuine "
    "misconceptions, so you arrive at an INCORRECT answer — but the reasoning stays "
    "coherent and internally consistent, the kind of honest mistake a real student "
    "makes. Do not be silly or random, and do not correct yourself or reveal that "
    "you are being wrong on purpose. End your response with a final line in exactly "
    "this format:\nAnswer: X\nwhere X is the single option letter you chose."
)

# user turn: "Question: <text>", then one "LETTER) option" per line, correct answer NOT shown
question = "Question: What is 1/2 + 1/3?\n\nA) 5/6\nB) 2/5\nC) 1/5\nD) 2/6"
msgs = [{"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": question}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# -> student reasoning, then a final line like:  Answer: B
```

For steadier per-item behavior, sample at a lower temperature (around 0.3); see `results/RESULTS.md` in
the repo. Local demo script: `python test_model.py`.

## Training

- **Base:** Qwen/Qwen3-1.7B (Instruct). **Method:** QLoRA (4-bit), LoRA r=16 / α=16, all 7 attn+MLP
  projections. **SFT:** 2 epochs, lr 2e-4 on ~3,421 examples. **DPO:** on top of SFT, β=0.1, 1 epoch,
  lr 5e-6, preference pairs (chosen = believable-wrong, rejected = correct-answer) to suppress the
  correctness prior. Trained on Colab (single GPU) via Unsloth/TRL.
- **Data:** see the dataset card. ~3.4k generated-and-filtered examples distilled from a frontier
  teacher; sources are Eedi / MalgoQA / ARC-Challenge / MMLU (formal_logic, logical_fallacies) / crafted math.

## Evaluation (base vs tuned, frozen 60-item held-out set, same system prompt)

| Metric | base | **v4-DPO** |
|---|---|---|
| Wrongness rate (chosen ≠ correct) | 20% | **73%** |
| by subject (math / science / logic) | 18/22/29% | **75/67/71%** |
| Character-break rate | 17% | **3%** |
| Spec adherence (0–2 judge) | 0.22 | **1.43** |
| Coherence (0–2 judge) | 1.58 | 1.50 |

A **prompted frontier model (claude-sonnet-4-6) scores 17%, on par with the untuned base (~20%)**, so the
fine-tune's behavior is not achievable by prompting. Robustness under adversarial attack and per-item
consistency are analyzed in `results/RESULTS.md`.

## Limitations

- **Wrongness is per-sample, not per-item:** at temperature 0.7 the model flips wrong↔right across
  resamples on ~60% of items (it's reliably *in character*, not reliably wrong on a given question).
  Lower temperature (≈0.3) improves per-item reliability.
- **Correctness prior partially reasserts on trivially-easy questions** (73% → ~50%).
- **~5% incoherent "wins"** historically: the answer letter not matching the reasoning's value. Guarded
  by `src/coherence.py` + a filter drop going forward; measured by `incoherent_win_rate`.
- Small model: do not expect frontier capability; that is not the goal.