M1 findings — a 1.5B SFT process-verifier hits an arithmetic-verification ceiling

Behavior: Model V, a generative process verifier that reads a problem + a step-indexed solution and returns the 0-based index of the first wrong step (-1 = all correct), scored on ProcessBench (first-error F1) with PRMBench as an adversarial check.

Headline: across two data recipes, more data, and higher LoRA rank, fine-tuning did not teach a Qwen2.5-Math-1.5B model to localize math errors on ProcessBench. It instead made the model more confident that solutions are correct. The failure is concentrated on arithmetic verification, which a 1.5B model cannot do reliably without a calculator — exactly why GenPRM-1.5B (the sub-4B precedent) needed code execution, a component we deliberately scoped out. This is the sub-4B fragility notes/research-small-vs-large-math.md warned about, observed directly.

Setup

Base: Qwen/Qwen2.5-Math-1.5B-Instruct (Apache-2.0, 4096 ctx). QLoRA, TRL 1.7.1 SFT.
Data: PRM800K first-error labels (MIT), critiques distilled from claude-opus-4-8, deduped vs ProcessBench/MATH-500. Two recipes (below).
Eval: ProcessBench (objective F1) + PRMBench (adversarial). Objective checkers, no LLM judge.
Baseline to beat: GPT-4o ProcessBench avg F1 = 61.9 (4-subset average).

Two data recipes tried

v1 — rationalized: teacher told the gold index, writes a consistent critique. 1,430 examples (rebalanced 50/50), LoRA r=16.
v2 — genuine detection: teacher critiques blind; keep a critique only if its predicted index matches the PRM800K gold (rejection sampling, 77% keep-rate). 8,127 examples (naturally ~50/50), LoRA r=32. This is the "distill the process, not the style" fix.

Results (ProcessBench gsm8k, 100 random examples)

model	decode	acc_error	acc_correct	F1	pred=−1 rate
base (untuned)	greedy	0.089	0.855	0.161	0.76
v1 (rationalized, 1.4k, r16)	greedy	0.111	0.945	0.199	0.83
v2 (genuine, 8.1k, r32)	greedy	0.089	0.982	0.163	0.92
v2	Maj@4	0.067	1.000	0.125	0.94

Reading it:

Fine-tuning raised the "all-correct" rate (0.76 → 0.83 → 0.92) — SFT taught the model to pass solutions more confidently, the opposite of a verifier's job. acc_error never moved off the base's ~0.09.
The "better" recipe was worse here: v2 (genuine detection, 5.7× data, 2× rank) collapsed to −1 more than v1 and scored below it on gsm8k. More/cleaner data did not help — a capability signal, not a data-quantity signal (cf. LoRA-Learns-Less: hard new skills need more than low-rank small-data SFT).
Maj@k hurts a non-confident verifier: sampling scatters the error-index votes while −1 stays the modal vote, so majority amplifies the −1 collapse (Maj@4 worse than greedy). Maj@k only helps once the base predictions are good (the GenPRM regime).

The bright spot that explains the failure — PRMBench (196 modified, Maj@1)

detection 0.413 (flags some error on 41% of erroneous items), localization 0.148, false-positive 0.205.
Per category: confidence 0.60, step_contradiction 0.50, domain_inconsistency 0.46, counterfactual 0.46, circular 0.39, missing_condition 0.39, deception 0.30, redundancy 0.26.

The same model that flags errors only ~8% of the time on gsm8k flags them 41% of the time on PRMBench — because PRMBench's injected errors are structural/logical (circular reasoning, step contradictions, confidence), which are detectable from language, whereas gsm8k errors are arithmetic, which a 1.5B model can't verify by "reading." V learned to catch the errors it can reason about and punts (→ −1) on the ones requiring computation.

Diagnosis (why SFT alone can't fix it)

Arithmetic-verification ceiling. 1.5B can't reliably recompute steps, so it hedges to "looks correct" on arithmetic-heavy gsm8k. GenPRM-1.5B needed code execution to reach 63 F1.
The decision is a needle in the critique. With completion_only_loss over ~400-token critiques, the boxed index is ~1 token; SFT optimizes critique prose (token-acc 0.78) and under-weights the actual verdict.
Distribution shift. Trained on MATH-based PRM800K (opus critiques); ProcessBench-gsm8k solutions come from other models with different, arithmetic-heavy error styles.

What this validates (the honest thesis)

The project's premise — "a small model can win on the easy side of the generator–verifier asymmetry" — holds only where the checkable signal is cheap (a compiler, a step label the model can read). For arithmetic step-checking, a 1.5B model without a calculator is on the hard side after all. This is a concrete, reproducible instance of the note's "sub-4B evidence is thin" caveat, with the mechanism identified.

Next steps (decided)

7B verifier — retrain V on Qwen2.5-Math-7B-Instruct (Apache, fits the 80GB A100). Tests whether raw capability alone lifts arithmetic verification. configs/verifier_v_7b.yaml.
Code execution (GenPRM's actual recipe) — let V run Python to check steps, targeting the arithmetic ceiling directly. The principled fix; bigger build.

Reproduce

results/processbench_base.json, processbench_tuned*.json, prmbench_tuned.json (this dir). Datasets: 42e/mathcompose-verifier (v2, genuine detection). Code: 42e/mathcompose.