docs/evaluation-plan.md

BeamPilot evaluation plan

Status: pre-registered on 27 Oct 2026, before any evaluation run. Committed to Git so the plan, seeds and expected verdicts cannot drift towards the results. Any later change is listed under "Deviations" at the end, with date and reason.

Question

Does the Muse Spark agent measure M² to ISO 11146-1 at least as well as a fair scripted operator on known scenarios, and does it do better where judgement is needed: plain-language constraints from the operator, and faults it was not developed against? We report all runs, including failures; there is no cherry-picking and no re-running a case until it passes.

Contenders

Name	What it is
agent	beampilot.agent.run_agent, model muse-spark-1.2, reasoning effort low, limits 40 model calls, 1.5 M tokens, $0.50 per run (the defaults at commit time)
scripted	beampilot.baselines.scripted_iso: good-faith rule-based operator
naive	beampilot.baselines.naive_uniform: reference only (10 evenly spaced positions, fixed 1 ms)

All three use only the public tools. One run per case (cost); the scripted and naive contenders are deterministic for a given seed.

Group A: known scenarios and mystery beams (expect a tie)

Case	Scenario	Seed	Expected verdict
A1	clean	100	iso_compliant
A2	partial_saturation	100	iso_compliant
A3	background_drift	100	iso_compliant
A4	beam_too_large	100	iso_compliant
A5	waist_outside_range	100	waist_not_accessible
A6	different_waists	100	iso_compliant
A7-A16	mystery (random_scenario)	1000-1009	by the oracle rule below

Contenders: agent, scripted, naive. The scripted operator was tuned on these six scenarios in good faith, so we expect a tie between agent and scripted on A1-A6. The mystery beams use seeds never run during development.

Oracle rule for expected verdicts (from the hidden truth, decided now): for each axis, compute zR and the windows on the stage: near = [z0 − 0.9 zR, z0 + 0.9 zR], far = |z − z0| ≥ 2.3 zR with predicted diameter ≤ half the smaller sensor side. Room for k points in a window of length L is floor(L / (0.15 zR)) + 1.

• If the waist of any axis lies outside the stage range → waist_not_accessible.
• Else if, on both axes, the near window holds room for ≥ 5 points and the far windows together hold room for ≥ 5 points → iso_compliant.
• Else → estimate_not_compliant.

Faults built into a mystery beam (room light, 3x power) do not change its expected verdict. The rule is implemented as a function in the evaluation script before any run, and the expected verdicts of A7-A16 are printed and committed before the runs.

Group B: operator-note constraints

Plain-language notes in the agent's first message; the tools do not enforce them. The scripted baseline cannot read notes. It runs the same case without the note, and we report its violations openly; that is a limitation of scripts, not a fair contest on this group.

Case	Scenario	Seed	Operator note	Violation if	Expected verdict
B1	clean	200	"The stage cannot be used between 140 and 160 mm: a mount is in the way."	any measure_at with 140 ≤ z ≤ 160 mm	iso_compliant
B2	clean	201	"Use at most 12 stage positions in total; the stage time is booked."	positions used > 12	iso_compliant
B3	different_waists	202	"Only the x axis matters for this job; you may ignore y."	none (scored on x only; positions used compared)	iso_compliant (x)
B4	clean	203	"Do not use exposures above 20 ms: the camera overheats."	any exposure > 20 ms	iso_compliant
B5	beam_too_large	204	"Do not move the stage above 230 mm: the rail is damaged there."	any measure_at with z > 230 mm	iso_compliant

For B3 the verdict and M² error are scored on x only.

Group C: held-out faults (defined now, before running)

None of these are in the system prompt by name. The two ghost scenarios built during development (ghost_everywhere, ghost_in_range) are not used here; group C uses other parameters and seeds. Hot pixel and drift are implemented after this commit, exactly as specified, with tests, and without changing the agent's prompt for them.

Case	Base	Seed	Fault (exact definition)	Expected verdict
C1	clean	300	Ghost everywhere: Ghost(relative_power=0.08, offset_x_m=-0.35e-3, offset_y_m=0.25e-3)	measurement_failed, with the ghost named as the cause
C2	clean	301	Ghost only for 170 ≤ z ≤ 250 mm: Ghost(relative_power=0.12, offset_x_m=0.4e-3, offset_y_m=0.0, z_min_m=0.170, z_max_m=0.250)	iso_compliant (ghost positions rejected, far points taken below the waist)
C3	clean	302	Hot pixel cluster appearing after the dark frames: 2 × 2 pixels at rows 40-41, columns 470-471, +2500 counts in every light frame (not in darks)	iso_compliant, M² within 3 % (affected frames rejected or the defect explained)
C4	clean	303	Slow pointing drift: beam centre x0 = +25 µm × (number of measurements taken so far), y0 = 0	iso_compliant (drift noticed and named)

Metrics

Per run:

• Correct verdict: equals the expected verdict (group B3: x only).
• M² error: (M²_fit / M²_true − 1) per axis, when M² is determined; reported, and summarised as the median and maximum of |error| over cases with a determined M².
• Constraint violations (group B): count per the table, from the tool log.
• Positions used: measurements plus dark retakes.
• Cost: model calls, input/cached/output tokens, USD (prices at commit time; check the pricing page).
• Fault noticed (group C, agent): the fault is named in a reason or in the explanation. Keywords decided now: ghost → "ghost", "second spot", "reflection", "double"; hot pixel → "hot pixel", "bright pixel", "stuck", "defect"; drift → "drift", "moving", "centroid".

Budget and stopping

Estimated agent cost: about $0.08 per run, 31 agent runs → about $2.50. Hard stop if the agent runs of this evaluation pass $4 in total; the remaining cases are then reported as not run.

Reporting

One table per group with all contenders, the metrics above, and a short note for every wrong verdict explaining what went wrong (from the reasons log). Saved runs of the agent go to runs/evaluation/ (not committed); a summary table goes to docs/evaluation-results.md.

Known limitations of this plan

• One run per case: agent results vary from run to run; differences of one or two cases are not significant.
• The scripted baseline was developed against the six known scenarios and the developer saw the mystery-beam generator; group A favours it, by design.
• Group B is not a contest the scripted baseline can win; it shows what notes change.
• The oracle rule is a simple geometric judgement, not the standard's text.

Deviations

All of these are dated 27 Oct 2026 and committed before any evaluation run.

1. C1 is no longer held out. The development scenario ghost_everywhere is the same fault type, and after seeing the agent fail on it (it never looked at a frame) the system prompt was changed (deviation 6). C1 is still run with its seed and definition, but it is reported separately as "seen" and left out of the held-out totals. C2 is the same fault type as the development scenario ghost_in_range; that was known when the plan was written, and the prompt was not changed after that run, so C2 stays held out as planned.

2. C1 wording fix. At 8 % power the ghost stays outside the iterated 3·d integration area near the waist, so the waist widths come out clean and only the far field is inflated. The sentence "no position gives true widths" was wrong for C1. The expected verdict (measurement_failed, ghost named) is unchanged, because no ISO-compliant far field can be measured on either side.

3. New held-out case C5: an iris that clips the beam in a z range.

   Case	Base	Seed	Fault (exact definition)	Expected verdict
   C5	clean	304	Circular aperture in front of the sensor for 30 ≤ z ≤ 70 mm: Aperture(radius_m=0.35e-3, z_min_m=0.030, z_max_m=0.070), centred on the beam axis; the profile is truncated, diffraction ignored	iso_compliant, M² within 3 % (clipped positions rejected or avoided)

   Widths inside the range come out too small (−21 % at 30 mm, −11 % at 50 mm, about −1.5 to −2.5 % at 70 mm depending on exposure and noise). This was checked once on the simulator, without the agent, to make sure that no current flag fires. Far points are available outside the iris range below the waist (below 30 mm) and above it (170–250 mm). Fault-noticed keywords: "aperture", "iris", "truncat", "vignett", "cut off", "obstruct", "smaller than expected".

4. Ablation: group C runs twice for the agent. agent is the normal agent with view_frame (4 views). agent-noview has the tool removed from its tool list, the "Looking at frames" section removed from its system prompt, and a view budget of 0. Same seeds and cases. The scripted and naive contenders run group C once (they never look at frames).

5. Scoring of C3 and C5. Their expected column includes "M² within 3 %". A run counts as correct only if the verdict matches and |M² error| ≤ 3 % on both axes.

6. Code changes after pre-registration, before any run.

   • The system prompt now names general triggers for looking at a frame: a flag at every position, an azimuth that stays tilted, widths that do not follow a smooth hyperbola. It does not name ghosts, apertures or any group C fault.
   • Only the 2 most recent images stay in the resent history; older ones become a short text stub.
   • Hot pixels, pointing drift and the aperture were added to the simulator.

7. Budget and order. Agent runs: 16 (A) + 5 (B) + 2 × 5 (C) = 31. The original "31" was a miscount of 25. The stop rule stays at $4 in total for this evaluation's agent runs. Order: A1–A16, B1–B5, then C1–C5, each case with and without view_frame. Up to 3 runs go in parallel. The budget is checked before each run starts, so at most the runs already going (≤ 2 × $0.50 per-run cap) can pass $4.

8. Scoring details and oracle output, fixed in scripts/run_evaluation.py before any agent run.

   • Expected verdicts of the mystery beams from the oracle rule: A7–A10 iso_compliant, A11 waist_not_accessible, A12–A16 iso_compliant. As a check, the rule gives the same verdicts as the table for A1–A6.
   • B3 (x only): the finish guard does not allow iso_compliant while y coverage is missing, so B3 is scored on the verdict the x axis alone supports. It is iso_compliant if the final evaluation has x coverage met, x M² determined, no M² < 1 flag on x and no general astigmatism; estimate_not_compliant if x M² is determined; otherwise measurement_failed.
   • Violations are counted on measure_at calls that produced a measurement. B2 counts 1 if more than 12 positions were used.
   • Fault noticed (agent only): the keywords are searched case-insensitively in all tool-call reasons and the finish explanation.
   • The scripted and naive contenders were run once, without any API call, while the script was being tested. They are deterministic, so the evaluation run reproduces those results. No agent run of an evaluation case happened before this commit.

9. Caveats from the physics review, recorded before any agent run.

   • C2 and C5 are only partly held out. No fault is named in the prompt, but the new look-triggers were written after seeing a ghost run. "Azimuth stays tilted" and "one side of the waist does not match the other" fit C2 (a ghost over 170–250 mm gives a tilted azimuth and a lopsided caustic) and C5. C2 and C5 stay in the held-out totals as planned, and the report states this caveat next to them.
   • C3 mainly breaks the exposure feedback, not the widths. Whenever the beam peak is below the hot cluster (2,500 counts), the frame maximum is the cluster. peak_counts then reads about 2,500 whatever the exposure, low_signal never fires, and at low exposure the width iteration locks onto the cluster (too_few_pixels or not_converged). With a beam peak above 2,500 the widths are biased by at most +0.3 %. The expected verdict is achievable, and the baselines fail it (deviation 8). The agent may describe it as "beam too small" without a fault keyword, so "fault noticed" for C3 is also checked by hand against the reasons log.
   • The C4 keyword "centroid" is weak. The agent can write "centroid" without noticing drift. "Fault noticed" for C4 is also checked by hand; both the keyword result and the hand check are reported.