| # NPC Reason — Pre-registration of lift criteria (FROZEN before any training) |
|
|
| Committed in Dispatch 1, before any SFT / distillation / RL exists. Same discipline as |
| BigBugAI v2: the success bars are fixed in advance so the eventual before/after delta means |
| something. These bars are deltas/targets chosen on principle; they do NOT depend on the |
| measured baseline values (recorded separately in `reports/baseline.md` / `BASELINE.lock`), |
| so there is no possibility of tuning the bars to the baseline. |
|
|
| The comparator for the primary bar is the base model's **FORMAT-PROMPT** number — the honest |
| question is "did training add something that prompting the base alone could not?", not |
| "did training beat a base that was never asked for the format?". |
|
|
| ## Metrics (all from the frozen mechanical verifier, VERIFIER.lock) |
|
|
| - **verifiable-rate** — fraction of chains that are mechanically VERIFIABLE |
| (≥1 load-bearing `<<EXPR=RESULT>>` assertion, all assertions verify, final answer composes). |
| - **accuracy** — fraction whose final answer equals gold (independent of verifiable). |
| - **verified-and-correct** — fraction that are BOTH verifiable AND correct. This is the headline. |
|
|
| ## Committed criteria |
|
|
| 1. **Primary (lift).** SFT NPC Reason must raise **verified-and-correct** by |
| **≥ +15 percentage points absolute** over the base model's **format-prompt** verified-and-correct rate. |
|
|
| 2. **Verifiable-rate target.** NPC Reason must reach **verifiable-rate ≥ 90%**. The project's |
| purpose is near-total mechanical grounding of load-bearing steps; a specialized model should |
| emit the checkable format almost always. |
|
|
| 3. **Accuracy guard (no regression trade).** NPC Reason **accuracy** must not regress by more |
| than **5 percentage points** versus the base model's **format-prompt** accuracy. Verifiability |
| must not be bought by sacrificing correctness — both axes matter. |
|
|
| SFT passes Dispatch 2 only if (1) AND (2) AND (3) all hold. |
|
|
| 4. **RL gating (later dispatch).** The RL phase is gated on SFT meeting criteria 1–3. RL's own |
| criteria will be pre-registered at that time, evaluated against a held-out slice never used |
| for SFT model selection. The verifier (VERIFIER.lock) is the RL reward and must remain |
| byte-identical. |
|
|
| ## Honesty clause |
|
|
| Report whatever the numbers say. A **null** — training fails to beat the format-prompt |
| baseline on the primary bar — is a valid, publishable finding, consistent with this being a |
| real experiment and not a foregone conclusion. The bars above are not moved after results are |
| seen; if they are missed, that is the result. |
|
|
| ## Frozen references |
|
|
| - Verifier: `verifier/VERIFIER.lock` |
| - Eval set: `eval/EVAL.lock` |
| - Baseline: `baseline/BASELINE.lock` |
| - This pre-registration: `reports/PREREG.lock` (sha256 of this file). |
|
|
