Upload folder using huggingface_hub

configs/verifier_v_7b.yaml

@@ -0,0 +1,29 @@
+# Model V on the 7B-Math base — tests whether raw capability lifts the
+# arithmetic-verification ceiling the 1.5B hit. Apache-2.0, fits an 80GB A100.
+# Run with --gpu-preset none so these (7B-sized) batch settings apply instead of
+# the 1.5B a100 preset:
+#   python -m mathcompose.train.train --task v --config configs/verifier_v_7b.yaml --gpu-preset none
+extends: "base.yaml"
+
+task: "v"
+output_dir: "runs/verifier_v_7b"
+hub_model_id: null            # e.g. "42e/mathcompose-verifier-7b"
+
+model:
+  base_id: "Qwen/Qwen2.5-Math-7B-Instruct"
+  max_context: 4096
+
+data:
+  train_path: "data/verifier/train.jsonl"   # same v2 genuine-detection data
+  val_path: "data/verifier/val.jsonl"
+
+train:
+  max_length: 4096
+  per_device_train_batch_size: 4   # 7B: smaller batch than the 1.5B preset
+  gradient_accumulation_steps: 4   # eff batch 16
+  gradient_checkpointing: true
+
+eval:
+  processbench_splits: ["gsm8k", "math", "olympiadbench", "omnimath"]
+  maj_k: 8
+  gpt4o_reference_f1: 61.9

results/M1_verifier_findings.md

@@ -0,0 +1,85 @@
+# M1 findings — a 1.5B SFT process-verifier hits an arithmetic-verification ceiling
+
+**Behavior:** Model V, a generative process verifier that reads a problem + a step-indexed
+solution and returns the 0-based index of the first wrong step (`-1` = all correct), scored on
+**ProcessBench** (first-error F1) with **PRMBench** as an adversarial check.
+
+**Headline:** across two data recipes, more data, and higher LoRA rank, fine-tuning **did not
+teach a Qwen2.5-Math-1.5B model to localize math errors** on ProcessBench. It instead made the
+model *more* confident that solutions are correct. The failure is concentrated on **arithmetic**
+verification, which a 1.5B model cannot do reliably without a calculator — exactly why GenPRM-1.5B
+(the sub-4B precedent) needed **code execution**, a component we deliberately scoped out. This is
+the sub-4B fragility `notes/research-small-vs-large-math.md` warned about, observed directly.
+
+## Setup
+- Base: `Qwen/Qwen2.5-Math-1.5B-Instruct` (Apache-2.0, 4096 ctx). QLoRA, TRL 1.7.1 SFT.
+- Data: PRM800K first-error labels (MIT), critiques distilled from `claude-opus-4-8`,
+  deduped vs ProcessBench/MATH-500. Two recipes (below).
+- Eval: ProcessBench (objective F1) + PRMBench (adversarial). Objective checkers, **no LLM judge**.
+- Baseline to beat: GPT-4o ProcessBench avg F1 = **61.9** (4-subset average).
+
+## Two data recipes tried
+- **v1 — rationalized:** teacher told the gold index, writes a consistent critique. 1,430 examples
+  (rebalanced 50/50), LoRA r=16.
+- **v2 — genuine detection:** teacher critiques *blind*; keep a critique only if its predicted
+  index matches the PRM800K gold (rejection sampling, **77% keep-rate**). 8,127 examples
+  (naturally ~50/50), LoRA r=32. This is the "distill the process, not the style" fix.
+
+## Results (ProcessBench gsm8k, 100 random examples)
+
+| model | decode | acc_error | acc_correct | F1 | pred=−1 rate |
+|---|---|---|---|---|---|
+| base (untuned) | greedy | 0.089 | 0.855 | 0.161 | **0.76** |
+| **v1** (rationalized, 1.4k, r16) | greedy | 0.111 | 0.945 | 0.199 | **0.83** |
+| **v2** (genuine, 8.1k, r32) | greedy | 0.089 | 0.982 | 0.163 | **0.92** |
+| v2 | Maj@4 | 0.067 | 1.000 | 0.125 | **0.94** |
+
+**Reading it:**
+- **Fine-tuning raised the "all-correct" rate (0.76 → 0.83 → 0.92)** — SFT taught the model to
+  *pass* solutions more confidently, the opposite of a verifier's job. `acc_error` never moved off
+  the base's ~0.09.
+- **The "better" recipe was worse here:** v2 (genuine detection, 5.7× data, 2× rank) collapsed to
+  −1 *more* than v1 and scored *below* it on gsm8k. More/cleaner data did not help — a capability
+  signal, not a data-quantity signal (cf. LoRA-Learns-Less: hard new skills need more than low-rank
+  small-data SFT).
+- **Maj@k hurts a non-confident verifier:** sampling scatters the error-index votes while −1 stays
+  the modal vote, so majority amplifies the −1 collapse (Maj@4 worse than greedy). Maj@k only helps
+  once the base predictions are good (the GenPRM regime).
+
+## The bright spot that explains the failure — PRMBench (196 modified, Maj@1)
+- **detection 0.413** (flags *some* error on 41% of erroneous items), localization 0.148,
+  false-positive 0.205.
+- Per category: **confidence 0.60, step_contradiction 0.50, domain_inconsistency 0.46,
+  counterfactual 0.46**, circular 0.39, missing_condition 0.39, deception 0.30, **redundancy 0.26**.
+
+The same model that flags errors only ~8% of the time on gsm8k flags them **41%** of the time on
+PRMBench — because PRMBench's injected errors are **structural/logical** (circular reasoning, step
+contradictions, confidence), which are detectable from language, whereas gsm8k errors are
+**arithmetic**, which a 1.5B model can't verify by "reading." V learned to catch the errors it can
+*reason about* and punts (→ −1) on the ones requiring *computation*.
+
+## Diagnosis (why SFT alone can't fix it)
+1. **Arithmetic-verification ceiling.** 1.5B can't reliably recompute steps, so it hedges to "looks
+   correct" on arithmetic-heavy gsm8k. GenPRM-1.5B needed **code execution** to reach 63 F1.
+2. **The decision is a needle in the critique.** With `completion_only_loss` over ~400-token
+   critiques, the boxed index is ~1 token; SFT optimizes critique *prose* (token-acc 0.78) and
+   under-weights the actual verdict.
+3. **Distribution shift.** Trained on MATH-based PRM800K (opus critiques); ProcessBench-gsm8k
+   solutions come from other models with different, arithmetic-heavy error styles.
+
+## What this validates (the honest thesis)
+The project's premise — "a small model can win on the *easy side* of the generator–verifier
+asymmetry" — holds **only where the checkable signal is cheap** (a compiler, a step label the model
+can *read*). For **arithmetic** step-checking, a 1.5B model without a calculator is on the *hard*
+side after all. This is a concrete, reproducible instance of the note's "sub-4B evidence is thin"
+caveat, with the mechanism identified.
+
+## Next steps (decided)
+1. **7B verifier** �� retrain V on `Qwen2.5-Math-7B-Instruct` (Apache, fits the 80GB A100). Tests
+   whether raw capability alone lifts arithmetic verification. `configs/verifier_v_7b.yaml`.
+2. **Code execution (GenPRM's actual recipe)** — let V run Python to check steps, targeting the
+   arithmetic ceiling directly. The principled fix; bigger build.
+
+## Reproduce
+`results/processbench_base.json`, `processbench_tuned*.json`, `prmbench_tuned.json` (this dir).
+Datasets: `42e/mathcompose-verifier` (v2, genuine detection). Code: `42e/mathcompose`.