NPC Reason — Baseline (untouched base, FROZEN)
Base: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B (MIT), untouched, greedy / deterministic
(temperature 0.0, max_tokens 12288, seed 0).
Eval: 500 problems (GSM8K test + MATH-500), EVAL.lock e1573cabaf440105….
Verifier: VERIFIER.lock d5d146cfbb9a69e1… (pure-code, frozen).
Generation wall-clock: 556.4s for 1000 chains (both prompts).
| plain-prompt | format-prompt | |
|---|---|---|
| verifiable-rate (%) | 0.0 | 0.0 |
| accuracy (%) | 59.8 | 61.6 |
| verified-and-correct (%) | 0.0 | 0.0 |
| mean assertions/chain | 0.0 | 0.0 |
Prompts (verbatim)
plain:
Solve this math problem. Show your work step by step, then give the final answer on the last line as \boxed{{ANSWER}}.
Problem: {problem}
format-requesting:
Solve this math problem. For EVERY load-bearing arithmetic step, write the computation as an inline checkable assertion in the exact form <<EXPR = RESULT>>, where EXPR is the arithmetic expression and RESULT is its value (for example <<3*8 = 24>>). If a quantity is reused, you may name it, e.g. let total = <<3*8 = 24>>, and reference it later as <<total + 6 = 30>>. Do not assert any number that drives the answer without wrapping it in <<...>>. End with the final answer as \boxed{{ANSWER}}, and make sure it equals the result of your last <<...>> step.
Problem: {problem}
Reading this honestly
- The format-prompt column is the comparator for training: it is the ceiling the base can reach by prompting alone. The pre-registered primary bar (+15pp verified-and-correct) is over THIS column, not over plain.
- The base's verifiable-rate is whatever the table says — measured, not assumed. If it is already high under format-prompt, the project's framing must reflect the real (smaller) delta.
- R1-Distill is a reasoning model; vendor recommends temp 0.6, but greedy is used here for a reproducible frozen baseline (repetition is possible; reported as-is).
This baseline is FROZEN (BASELINE.lock). Every training dispatch compares against exactly these numbers.
Greedy no-answer note (honest)
83/500 format-prompt chains (and a similar share of plain) produced NO extractable final answer within the 12,288-token budget — these are R1-Distill greedy repetition loops (the vendor recommends temperature 0.6 for exactly this reason). We keep GREEDY for a reproducible frozen baseline and count these as wrong; accuracy here is therefore a FLOOR. Raising the budget 4096->12288 only moved no-answer 95->83, confirming the cause is degeneration, not truncation. The SAME greedy decoding will be applied to NPC Reason after training, so the before/after comparison is apples-to-apples (and training is expected to reduce the looping). A prior 4096-token run is archived at baseline/baseline_results.trunc4096.json.