Blame for docs/evaluation-plan-v2.md

# BeamPilot evaluation plan v2

Status: pre-registered on 27 Oct 2026, **before any v2 run and before the code

changes it tests**. This file is the first commit on branch `add-agent-efficiency`.
Any later change goes under "Deviations" at the end, with date and reason.

v1 ([evaluation-plan.md](evaluation-plan.md), results in
[evaluation-results.md](evaluation-results.md)) **stays unchanged**. v2 does not
replace it; v2 results go to `docs/evaluation-results-v2.md`.

## Why v2

v1 showed an efficiency problem, not a judgement problem:
- **A8 and C2 were stopped by the 40-model-call limit** before `finish`; the
  controller's fallback verdict made both wrong.
- **In 12 of 31 agent runs the model made one tool call per model call** instead of
  requesting independent calls together.

v2 makes exactly two changes to the agent and nothing else (tools, simulator,
scoring, scripted and naive contenders all stay as in v1):

1. **Model-call limit 40 → 60.** The token budget (1.5 M per run) and the cost cap
   ($0.50 per run) stay unchanged.
2. **Prompt:** it says clearly that independent measurements are to be requested
   together in one turn (parallel tool calls), and that after each `evaluate` the
   agent plans several positions at once. The prompt stays under 1,500 words and
   still names no fault.

The question: with these changes, does the agent finish on its own in every run,
with fewer model calls, and without losing accuracy?

## Contenders

| Name | What it is |
|---|---|
| `agent` | `run_agent`, model `muse-spark-1.2`, reasoning effort `low`, limits **60 model calls**, 1.5 M tokens, $0.50 per run |
| `agent-noview` | as `agent`, with `view_frame` switched off (group C only, as v1 deviation 4) |
| `scripted` | `scripted_iso`, unchanged |
| `naive` | `naive_uniform`, unchanged (reference only) |

## Cases: same groups and definitions as v1, new seeds

**New seeds.** None of these seeds was used before. Used so far: 0–99 (tests and
development runs), 100, 200–204, 300–304 and 1000–1009 (v1).

| Group | Cases | Definition | Seeds (v1 → v2) |
|---|---|---|---|
| A | A1–A6 | the six known scenarios | 100 → **500** |
| A | A7–A16 | mystery beams (`random_scenario`) | 1000–1009 → **3000–3009** |
| B | B1–B5 | operator notes, word for word as in v1 | 200–204 → **600–604** |
| C | C1–C5 | the five faults exactly as defined in v1 (incl. C5 from deviation 3) | 300–304 → **700–704** |

The seed sets only the noise for A1–A6, B and C, so their expected verdicts are
the same as in v1. Expected verdicts of the new mystery beams, from v1's oracle
rule (computed before this commit, without any API call):

| Case | Seed | Expected | Case | Seed | Expected |
|---|---|---|---|---|---|
| A7 | 3000 | iso_compliant | A12 | 3005 | iso_compliant |
| A8 | 3001 | iso_compliant | A13 | 3006 | iso_compliant |
| A9 | 3002 | iso_compliant | A14 | 3007 | iso_compliant |
| A10 | 3003 | iso_compliant | A15 | 3008 | waist_not_accessible |

| A11 | 3004 | iso_compliant | A16 | 3009 | iso_compliant |



**Nothing in group C is held out in v2.** All five fault types were seen in the v1

results (C1 also during development). v2 measures efficiency and repeatability on

new noise and new mystery beams, not generalisation to new faults. Group C totals

are reported as "seen in v1".



## Metrics



The same as v1, with the same scoring rules (v1 deviations 5 and 8):

- correct verdict, with B3 scored on x only;

- C3 and C5 count as correct only if M² is within 3 %;

- M² error;

- constraint violations in group B;

- positions used;

- cost;

- fault noticed (keywords, plus the hand check for C3 and C4).



Also reported, not scored:

- model calls per run;

- tool calls per model call (batching);

- the number of runs stopped by a limit.



## Comparison with v1



`docs/evaluation-results-v2.md` gets, next to the v2 tables, a short table for

`agent` v1 against v2:

- correct verdicts per group;

- mean positions;

- mean model calls;

- tool calls per model call;

- runs stopped by the call limit;

- median and maximum |M² error|;

- cost.



The seeds differ between v1 and v2 and each case runs once. A difference of one or

two cases is therefore noise, not a measured effect.



## Budget and stopping



Agent runs: 16 (A) + 5 (B) + 2 × 5 (C) = 31.

- **Hard stop at $4** in total for the v2 agent runs; the remaining cases are then

  reported as not run.

- Order: A1–A16, B1–B5, then C1–C5, each case with and without `view_frame`.
- Up to 3 runs go in parallel. The budget is checked before each run starts.
- No code changes between runs. If something breaks, the evaluation stops and is
  reported, and no case is re-run.

## Known limitations

- One run per case, as in v1.
- Two changes at once (call limit and prompt). If v2 improves, this plan cannot say
  which change caused it. The model-call and batching numbers give a hint.
- A higher call limit can raise the cost per run; the per-run cap stays at $0.50.

## Deviations

Both dated 27 Oct 2026 and committed **before any v2 run**, after the physics
review of the v2 changes.

1. **The prompt changes more than the two lines described above.** To stay under
   1,500 words, three sentences were also shortened or dropped:
   - the frames-per-position rule;
   - the saturation hint;
   - "You explain every decision briefly."

   The required `reason` argument is still "one short sentence", but shorter reasons
   could lower the keyword hits for "fault noticed" in group C, independently of
   the two intended changes.
2. **Wording fix: no accept before seeing a result.** The Fill step said "Then accept
   or reject the results and call `evaluate` in the same turn". Measurement ids are
   predictable (m1, m2, …), so this could be read as accepting a batch in the same
   turn that measures it, without seeing the flags. It now says "Once you have seen
   the results, accept or reject them and call `evaluate` in that same turn". The
   v2 report counts any `set_status` that lands in the same model call as the
   `measure_at` that created its id.