results/M1_verifier_findings.md

# M1 findings — a 1.5B SFT process-verifier hits an arithmetic-verification ceiling

**Behavior:** Model V, a generative process verifier that reads a problem + a step-indexed
solution and returns the 0-based index of the first wrong step (`-1` = all correct), scored on
**ProcessBench** (first-error F1) with **PRMBench** as an adversarial check.

**Headline:** across two data recipes, more data, and higher LoRA rank, fine-tuning **did not
teach a Qwen2.5-Math-1.5B model to localize math errors** on ProcessBench. It instead made the
model *more* confident that solutions are correct. The failure is concentrated on **arithmetic**
verification, which a 1.5B model cannot do reliably without a calculator — exactly why GenPRM-1.5B
(the sub-4B precedent) needed **code execution**, a component we deliberately scoped out. This is
the sub-4B fragility `notes/research-small-vs-large-math.md` warned about, observed directly.

## Setup
- Base: `Qwen/Qwen2.5-Math-1.5B-Instruct` (Apache-2.0, 4096 ctx). QLoRA, TRL 1.7.1 SFT.
- Data: PRM800K first-error labels (MIT), critiques distilled from `claude-opus-4-8`,
  deduped vs ProcessBench/MATH-500. Two recipes (below).
- Eval: ProcessBench (objective F1) + PRMBench (adversarial). Objective checkers, **no LLM judge**.
- Baseline to beat: GPT-4o ProcessBench avg F1 = **61.9** (4-subset average).

## Two data recipes tried
- **v1 — rationalized:** teacher told the gold index, writes a consistent critique. 1,430 examples
  (rebalanced 50/50), LoRA r=16.
- **v2 — genuine detection:** teacher critiques *blind*; keep a critique only if its predicted
  index matches the PRM800K gold (rejection sampling, **77% keep-rate**). 8,127 examples
  (naturally ~50/50), LoRA r=32. This is the "distill the process, not the style" fix.

## Results (ProcessBench gsm8k, 100 random examples)

| model | decode | acc_error | acc_correct | F1 | pred=−1 rate |
|---|---|---|---|---|---|
| base (untuned) | greedy | 0.089 | 0.855 | 0.161 | **0.76** |
| **v1** (rationalized, 1.4k, r16) | greedy | 0.111 | 0.945 | 0.199 | **0.83** |
| **v2** (genuine, 8.1k, r32) | greedy | 0.089 | 0.982 | 0.163 | **0.92** |
| v2 | Maj@4 | 0.067 | 1.000 | 0.125 | **0.94** |

**Reading it:**
- **Fine-tuning raised the "all-correct" rate (0.76 → 0.83 → 0.92)** — SFT taught the model to
  *pass* solutions more confidently, the opposite of a verifier's job. `acc_error` never moved off
  the base's ~0.09.
- **The "better" recipe was worse here:** v2 (genuine detection, 5.7× data, 2× rank) collapsed to
  −1 *more* than v1 and scored *below* it on gsm8k. More/cleaner data did not help — a capability
  signal, not a data-quantity signal (cf. LoRA-Learns-Less: hard new skills need more than low-rank
  small-data SFT).
- **Maj@k hurts a non-confident verifier:** sampling scatters the error-index votes while −1 stays
  the modal vote, so majority amplifies the −1 collapse (Maj@4 worse than greedy). Maj@k only helps
  once the base predictions are good (the GenPRM regime).

## The bright spot that explains the failure — PRMBench (196 modified, Maj@1)
- **detection 0.413** (flags *some* error on 41% of erroneous items), localization 0.148,
  false-positive 0.205.
- Per category: **confidence 0.60, step_contradiction 0.50, domain_inconsistency 0.46,
  counterfactual 0.46**, circular 0.39, missing_condition 0.39, deception 0.30, **redundancy 0.26**.

The same model that flags errors only ~8% of the time on gsm8k flags them **41%** of the time on
PRMBench — because PRMBench's injected errors are **structural/logical** (circular reasoning, step
contradictions, confidence), which are detectable from language, whereas gsm8k errors are
**arithmetic**, which a 1.5B model can't verify by "reading." V learned to catch the errors it can
*reason about* and punts (→ −1) on the ones requiring *computation*.

## Diagnosis (why SFT alone can't fix it)
1. **Arithmetic-verification ceiling.** 1.5B can't reliably recompute steps, so it hedges to "looks
   correct" on arithmetic-heavy gsm8k. GenPRM-1.5B needed **code execution** to reach 63 F1.
2. **The decision is a needle in the critique.** With `completion_only_loss` over ~400-token
   critiques, the boxed index is ~1 token; SFT optimizes critique *prose* (token-acc 0.78) and
   under-weights the actual verdict.
3. **Distribution shift.** Trained on MATH-based PRM800K (opus critiques); ProcessBench-gsm8k
   solutions come from other models with different, arithmetic-heavy error styles.

## What this validates (the honest thesis)
The project's premise — "a small model can win on the *easy side* of the generator–verifier
asymmetry" — holds **only where the checkable signal is cheap** (a compiler, a step label the model
can *read*). For **arithmetic** step-checking, a 1.5B model without a calculator is on the *hard*
side after all. This is a concrete, reproducible instance of the note's "sub-4B evidence is thin"
caveat, with the mechanism identified.

## Next steps (decided)
1. **7B verifier** — retrain V on `Qwen2.5-Math-7B-Instruct` (Apache, fits the 80GB A100). Tests
   whether raw capability alone lifts arithmetic verification. `configs/verifier_v_7b.yaml`.
2. **Code execution (GenPRM's actual recipe)** — let V run Python to check steps, targeting the
   arithmetic ceiling directly. The principled fix; bigger build.

## Reproduce
`results/processbench_base.json`, `processbench_tuned*.json`, `prmbench_tuned.json` (this dir).
Datasets: `42e/mathcompose-verifier` (v2, genuine detection). Code: `42e/mathcompose`.