docs/evaluation-plan-v2.md

BeamPilot evaluation plan v2

Status: pre-registered on 27 Oct 2026, before any v2 run and before the code changes it tests. This file is the first commit on branch add-agent-efficiency. Any later change goes under "Deviations" at the end, with date and reason.

v1 (evaluation-plan.md, results in evaluation-results.md) stays unchanged. v2 does not replace it; v2 results go to docs/evaluation-results-v2.md.

Why v2

v1 showed an efficiency problem, not a judgement problem:

• A8 and C2 were stopped by the 40-model-call limit before finish; the controller's fallback verdict made both wrong.
• In 12 of 31 agent runs the model made one tool call per model call instead of requesting independent calls together.

v2 makes exactly two changes to the agent and nothing else (tools, simulator, scoring, scripted and naive contenders all stay as in v1):

1. Model-call limit 40 → 60. The token budget (1.5 M per run) and the cost cap ($0.50 per run) stay unchanged.
2. Prompt: it says clearly that independent measurements are to be requested together in one turn (parallel tool calls), and that after each evaluate the agent plans several positions at once. The prompt stays under 1,500 words and still names no fault.

The question: with these changes, does the agent finish on its own in every run, with fewer model calls, and without losing accuracy?

Contenders

Name	What it is
agent	run_agent, model muse-spark-1.2, reasoning effort low, limits 60 model calls, 1.5 M tokens, $0.50 per run
agent-noview	as agent, with view_frame switched off (group C only, as v1 deviation 4)
scripted	scripted_iso, unchanged
naive	naive_uniform, unchanged (reference only)

Cases: same groups and definitions as v1, new seeds

New seeds. None of these seeds was used before. Used so far: 0–99 (tests and development runs), 100, 200–204, 300–304 and 1000–1009 (v1).

Group	Cases	Definition	Seeds (v1 → v2)
A	A1–A6	the six known scenarios	100 → 500
A	A7–A16	mystery beams (random_scenario)	1000–1009 → 3000–3009
B	B1–B5	operator notes, word for word as in v1	200–204 → 600–604
C	C1–C5	the five faults exactly as defined in v1 (incl. C5 from deviation 3)	300–304 → 700–704

The seed sets only the noise for A1–A6, B and C, so their expected verdicts are the same as in v1. Expected verdicts of the new mystery beams, from v1's oracle rule (computed before this commit, without any API call):

Case	Seed	Expected	Case	Seed	Expected
A7	3000	iso_compliant	A12	3005	iso_compliant
A8	3001	iso_compliant	A13	3006	iso_compliant
A9	3002	iso_compliant	A14	3007	iso_compliant
A10	3003	iso_compliant	A15	3008	waist_not_accessible
A11	3004	iso_compliant	A16	3009	iso_compliant

Nothing in group C is held out in v2. All five fault types were seen in the v1 results (C1 also during development). v2 measures efficiency and repeatability on new noise and new mystery beams, not generalisation to new faults. Group C totals are reported as "seen in v1".

Metrics

The same as v1, with the same scoring rules (v1 deviations 5 and 8):

• correct verdict, with B3 scored on x only;
• C3 and C5 count as correct only if M² is within 3 %;
• M² error;
• constraint violations in group B;
• positions used;
• cost;
• fault noticed (keywords, plus the hand check for C3 and C4).

Also reported, not scored:

• model calls per run;
• tool calls per model call (batching);
• the number of runs stopped by a limit.

Comparison with v1

docs/evaluation-results-v2.md gets, next to the v2 tables, a short table for agent v1 against v2:

• correct verdicts per group;
• mean positions;
• mean model calls;
• tool calls per model call;
• runs stopped by the call limit;
• median and maximum |M² error|;
• cost.

The seeds differ between v1 and v2 and each case runs once. A difference of one or two cases is therefore noise, not a measured effect.

Budget and stopping

Agent runs: 16 (A) + 5 (B) + 2 × 5 (C) = 31.

• Hard stop at $4 in total for the v2 agent runs; the remaining cases are then reported as not run.
• Order: A1–A16, B1–B5, then C1–C5, each case with and without view_frame.
• Up to 3 runs go in parallel. The budget is checked before each run starts.
• No code changes between runs. If something breaks, the evaluation stops and is reported, and no case is re-run.

Known limitations

• One run per case, as in v1.
• Two changes at once (call limit and prompt). If v2 improves, this plan cannot say which change caused it. The model-call and batching numbers give a hint.
• A higher call limit can raise the cost per run; the per-run cap stays at $0.50.

Deviations

Both dated 27 Oct 2026 and committed before any v2 run, after the physics review of the v2 changes.

1. The prompt changes more than the two lines described above. To stay under 1,500 words, three sentences were also shortened or dropped:

   • the frames-per-position rule;
   • the saturation hint;
   • "You explain every decision briefly."

   The required reason argument is still "one short sentence", but shorter reasons could lower the keyword hits for "fault noticed" in group C, independently of the two intended changes.

2. Wording fix: no accept before seeing a result. The Fill step said "Then accept or reject the results and call evaluate in the same turn". Measurement ids are predictable (m1, m2, …), so this could be read as accepting a batch in the same turn that measures it, without seeing the flags. It now says "Once you have seen the results, accept or reject them and call evaluate in that same turn". The v2 report counts any set_status that lands in the same model call as the measure_at that created its id.