=== NPC Reason — Dispatch 1: Verifier + Frozen Baseline ===
Verifier: mechanical <<EXPR=RESULT>> checker (SymPy), pure-code, no LLM judgment. VERIFIER.lock d5d146cf… (also the later RL reward signal). tests: correct / arithmetic-error / fluency-trap / non-composing / verifiable-but-wrong / correct-but-unverifiable / variable-binding / unbound-fail-closed / tolerance(exact+1e-6) / extraction — 14/14 pass. Eval: 500 problems = 300 GSM8K(test) + 200 MATH-500, held-out & frozen. EVAL.lock e1573cab… ; meta records sources/splits/seed(20260615). Never training data. Baseline: DeepSeek-R1-Distill-Qwen-1.5B (MIT), untouched, greedy/deterministic (seed 0, max_tokens 12288, ctx 16384). 1000 chains in 556s. BASELINE.lock frozen.
plain-prompt format-prompt
verifiable-rate 0.0% 0.0% <-- base never emits <<...>> (1/500 contain "<<")
accuracy 59.8% 61.6% (floor; greedy repetition leaves 83/500 fmt no-answer)
verified-and-correct 0.0% 0.0%
mean assertions/chain 0.0 0.0
Reading: The base does NOT produce mechanically-checkable steps, even when explicitly asked
(format-prompt verifiable-rate is 0%, not just plain). The project's headline delta
is therefore real and large-headroom: any verifiable-rate NPC Reason reaches is lift
over 0. Accuracy is a separate axis (60%) and is the guard, not the goal.
Prereg: FROZEN before training (PREREG.lock b5a49437…). Bars committed: 1. primary: verified-and-correct +>=15pp over format baseline (i.e. >=15%) 2. verifiable-rate >= 90% 3. accuracy must not regress > 5pp below format accuracy 61.6% (i.e. >= 56.6%) 4. RL gated on SFT passing 1-3; RL criteria pre-registered later. Honesty clause: a null (training fails to beat format baseline) is a valid finding.
Next: Dispatch 2 = generate DeepSeek-V4 verifiable <<...>> chains, code-filter against VERIFIER.lock, SFT-distill into R1-Distill-1.5B; decontaminate against EVAL.lock. GPU: inference only; A40 ~4.5GB peak. No training performed this dispatch.
Notes: - Re-ran baseline at 12288 tokens (from 4096) to remove a truncation artifact; 4096 run archived at baseline/baseline_results.trunc4096.json. verifiable-rate 0% in BOTH runs (robust). Verifier frozen BEFORE both baseline runs (not tuned to them). - R1-Distill greedy repetition (vendor recommends temp 0.6) leaves ~17% of chains without a final answer; counted wrong, so accuracy is a floor. Same decoding will apply post-training -> apples-to-apples.