File size: 2,833 Bytes
1ba301a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | === NPC Reason — Dispatch 1: Verifier + Frozen Baseline ===
Verifier: mechanical <<EXPR=RESULT>> checker (SymPy), pure-code, no LLM judgment.
VERIFIER.lock d5d146cf… (also the later RL reward signal).
tests: correct / arithmetic-error / fluency-trap / non-composing /
verifiable-but-wrong / correct-but-unverifiable / variable-binding /
unbound-fail-closed / tolerance(exact+1e-6) / extraction — 14/14 pass.
Eval: 500 problems = 300 GSM8K(test) + 200 MATH-500, held-out & frozen.
EVAL.lock e1573cab… ; meta records sources/splits/seed(20260615). Never training data.
Baseline: DeepSeek-R1-Distill-Qwen-1.5B (MIT), untouched, greedy/deterministic (seed 0,
max_tokens 12288, ctx 16384). 1000 chains in 556s. BASELINE.lock frozen.
plain-prompt format-prompt
verifiable-rate 0.0% 0.0% <-- base never emits <<...>> (1/500 contain "<<")
accuracy 59.8% 61.6% (floor; greedy repetition leaves 83/500 fmt no-answer)
verified-and-correct 0.0% 0.0%
mean assertions/chain 0.0 0.0
Reading: The base does NOT produce mechanically-checkable steps, even when explicitly asked
(format-prompt verifiable-rate is 0%, not just plain). The project's headline delta
is therefore real and large-headroom: any verifiable-rate NPC Reason reaches is lift
over ~0. Accuracy is a separate axis (~60%) and is the guard, not the goal.
Prereg: FROZEN before training (PREREG.lock b5a49437…). Bars committed:
1. primary: verified-and-correct +>=15pp over format baseline (i.e. >=15%)
2. verifiable-rate >= 90%
3. accuracy must not regress > 5pp below format accuracy 61.6% (i.e. >= 56.6%)
4. RL gated on SFT passing 1-3; RL criteria pre-registered later.
Honesty clause: a null (training fails to beat format baseline) is a valid finding.
Next: Dispatch 2 = generate DeepSeek-V4 verifiable <<...>> chains, code-filter against
VERIFIER.lock, SFT-distill into R1-Distill-1.5B; decontaminate against EVAL.lock.
GPU: inference only; A40 ~4.5GB peak. No training performed this dispatch.
Notes: - Re-ran baseline at 12288 tokens (from 4096) to remove a truncation artifact;
4096 run archived at baseline/baseline_results.trunc4096.json. verifiable-rate
0% in BOTH runs (robust). Verifier frozen BEFORE both baseline runs (not tuned to them).
- R1-Distill greedy repetition (vendor recommends temp 0.6) leaves ~17% of chains
without a final answer; counted wrong, so accuracy is a floor. Same decoding will
apply post-training -> apples-to-apples.
|