NPC Reason — Pre-registration of lift criteria (FROZEN before any training)
Committed in Dispatch 1, before any SFT / distillation / RL exists. Same discipline as
BigBugAI v2: the success bars are fixed in advance so the eventual before/after delta means
something. These bars are deltas/targets chosen on principle; they do NOT depend on the
measured baseline values (recorded separately in reports/baseline.md / BASELINE.lock),
so there is no possibility of tuning the bars to the baseline.
The comparator for the primary bar is the base model's FORMAT-PROMPT number — the honest question is "did training add something that prompting the base alone could not?", not "did training beat a base that was never asked for the format?".
Metrics (all from the frozen mechanical verifier, VERIFIER.lock)
- verifiable-rate — fraction of chains that are mechanically VERIFIABLE
(≥1 load-bearing
<<EXPR=RESULT>>assertion, all assertions verify, final answer composes). - accuracy — fraction whose final answer equals gold (independent of verifiable).
- verified-and-correct — fraction that are BOTH verifiable AND correct. This is the headline.
Committed criteria
Primary (lift). SFT NPC Reason must raise verified-and-correct by ≥ +15 percentage points absolute over the base model's format-prompt verified-and-correct rate.
Verifiable-rate target. NPC Reason must reach verifiable-rate ≥ 90%. The project's purpose is near-total mechanical grounding of load-bearing steps; a specialized model should emit the checkable format almost always.
Accuracy guard (no regression trade). NPC Reason accuracy must not regress by more than 5 percentage points versus the base model's format-prompt accuracy. Verifiability must not be bought by sacrificing correctness — both axes matter.
SFT passes Dispatch 2 only if (1) AND (2) AND (3) all hold.
RL gating (later dispatch). The RL phase is gated on SFT meeting criteria 1–3. RL's own criteria will be pre-registered at that time, evaluated against a held-out slice never used for SFT model selection. The verifier (VERIFIER.lock) is the RL reward and must remain byte-identical.
Honesty clause
Report whatever the numbers say. A null — training fails to beat the format-prompt baseline on the primary bar — is a valid, publishable finding, consistent with this being a real experiment and not a foregone conclusion. The bars above are not moved after results are seen; if they are missed, that is the result.
Frozen references
- Verifier:
verifier/VERIFIER.lock - Eval set:
eval/EVAL.lock - Baseline:
baseline/BASELINE.lock - This pre-registration:
reports/PREREG.lock(sha256 of this file).