reports/baseline.md

# NPC Reason — Baseline (untouched base, FROZEN)

Base: **deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B** (MIT), untouched, **greedy / deterministic**
(temperature 0.0, max_tokens 12288, seed 0).
Eval: **500** problems (GSM8K test + MATH-500), EVAL.lock `e1573cabaf440105…`.
Verifier: VERIFIER.lock `d5d146cfbb9a69e1…` (pure-code, frozen).
Generation wall-clock: 556.4s for 1000 chains (both prompts).

|                          |   plain-prompt |  format-prompt |
|--------------------------|---------------:|---------------:|
| verifiable-rate (%)      |          0.0 |           0.0 |
| accuracy (%)             |         59.8 |          61.6 |
| verified-and-correct (%) |          0.0 |           0.0 |
| mean assertions/chain    |          0.0 |           0.0 |

## Prompts (verbatim)

**plain:**
```
Solve this math problem. Show your work step by step, then give the final answer on the last line as \boxed{{ANSWER}}.

Problem: {problem}
```

**format-requesting:**
```
Solve this math problem. For EVERY load-bearing arithmetic step, write the computation as an inline checkable assertion in the exact form <<EXPR = RESULT>>, where EXPR is the arithmetic expression and RESULT is its value (for example <<3*8 = 24>>). If a quantity is reused, you may name it, e.g. let total = <<3*8 = 24>>, and reference it later as <<total + 6 = 30>>. Do not assert any number that drives the answer without wrapping it in <<...>>. End with the final answer as \boxed{{ANSWER}}, and make sure it equals the result of your last <<...>> step.

Problem: {problem}
```

## Reading this honestly

- The **format-prompt** column is the comparator for training: it is the ceiling the base can
  reach by prompting alone. The pre-registered primary bar (+15pp verified-and-correct) is over
  THIS column, not over plain.
- The base's verifiable-rate is whatever the table says — measured, not assumed. If it is already
  high under format-prompt, the project's framing must reflect the real (smaller) delta.
- R1-Distill is a reasoning model; vendor recommends temp 0.6, but greedy is used here for a reproducible frozen baseline (repetition is possible; reported as-is).

This baseline is FROZEN (BASELINE.lock). Every training dispatch compares against exactly these numbers.

## Greedy no-answer note (honest)

83/500 format-prompt chains (and a similar share of plain) produced NO extractable final
answer within the 12,288-token budget — these are R1-Distill greedy **repetition loops**
(the vendor recommends temperature 0.6 for exactly this reason). We keep GREEDY for a
reproducible frozen baseline and count these as wrong; accuracy here is therefore a FLOOR.
Raising the budget 4096->12288 only moved no-answer 95->83, confirming the cause is
degeneration, not truncation. The SAME greedy decoding will be applied to NPC Reason after
training, so the before/after comparison is apples-to-apples (and training is expected to
reduce the looping). A prior 4096-token run is archived at baseline/baseline_results.trunc4096.json.