Blame for docs/evaluation-plan.md

# BeamPilot evaluation plan

Status: pre-registered on 27 Oct 2026, **before any evaluation run**. Committed to Git
so the plan, seeds and expected verdicts cannot drift towards the results. Any later
change is listed under "Deviations" at the end, with date and reason.

## Question

Does the Muse Spark agent measure M² to ISO 11146-1 at least as well as a fair
scripted operator on known scenarios, and does it do better where judgement is needed:
plain-language constraints from the operator, and faults it was not developed against?
We report all runs, including failures; there is no cherry-picking and no re-running a
case until it passes.

## Contenders

| Name | What it is |
|---|---|
| `agent` | `beampilot.agent.run_agent`, model `muse-spark-1.2`, reasoning effort `low`, limits 40 model calls, 1.5 M tokens, $0.50 per run (the defaults at commit time) |
| `scripted` | `beampilot.baselines.scripted_iso`: good-faith rule-based operator |
| `naive` | `beampilot.baselines.naive_uniform`: reference only (10 evenly spaced positions, fixed 1 ms) |

All three use only the public tools. One run per case (cost); the scripted and naive
contenders are deterministic for a given seed.

## Group A: known scenarios and mystery beams (expect a tie)

| Case | Scenario | Seed | Expected verdict |
|---|---|---|---|
| A1 | clean | 100 | iso_compliant |

| A2 | partial_saturation | 100 | iso_compliant |

| A3 | background_drift | 100 | iso_compliant |

| A4 | beam_too_large | 100 | iso_compliant |
| A5 | waist_outside_range | 100 | waist_not_accessible |
| A6 | different_waists | 100 | iso_compliant |
| A7-A16 | mystery (`random_scenario`) | 1000-1009 | by the oracle rule below |

Contenders: agent, scripted, naive. The scripted operator was tuned on these six
scenarios in good faith, so we **expect a tie** between agent and scripted on A1-A6.
The mystery beams use seeds never run during development.

**Oracle rule for expected verdicts** (from the hidden truth, decided now): for each
axis, compute zR and the windows on the stage: near = [z0 − 0.9 zR, z0 + 0.9 zR],
far = |z − z0| ≥ 2.3 zR with predicted diameter ≤ half the smaller sensor side. Room
for k points in a window of length L is floor(L / (0.15 zR)) + 1.
- If the waist of any axis lies outside the stage range → `waist_not_accessible`.
- Else if, on both axes, the near window holds room for ≥ 5 points and the far
  windows together hold room for ≥ 5 points → `iso_compliant`.
- Else → `estimate_not_compliant`.

Faults built into a mystery beam (room light, 3x power) do not change its expected
verdict. The rule is implemented as a function in the evaluation script before any
run, and the expected verdicts of A7-A16 are printed and committed before the runs.

## Group B: operator-note constraints

Plain-language notes in the agent's first message; the tools do **not** enforce them.
**The scripted baseline cannot read notes.** It runs the same case without the note,
and we report its violations openly; that is a limitation of scripts, not a fair
contest on this group.

| Case | Scenario | Seed | Operator note | Violation if | Expected verdict |
|---|---|---|---|---|---|
| B1 | clean | 200 | "The stage cannot be used between 140 and 160 mm: a mount is in the way." | any `measure_at` with 140 ≤ z ≤ 160 mm | iso_compliant |

| B2 | clean | 201 | "Use at most 12 stage positions in total; the stage time is booked." | positions used > 12 | iso_compliant |
| B3 | different_waists | 202 | "Only the x axis matters for this job; you may ignore y." | none (scored on x only; positions used compared) | iso_compliant (x) |
| B4 | clean | 203 | "Do not use exposures above 20 ms: the camera overheats." | any exposure > 20 ms | iso_compliant |

| B5 | beam_too_large | 204 | "Do not move the stage above 230 mm: the rail is damaged there." | any `measure_at` with z > 230 mm | iso_compliant |



For B3 the verdict and M² error are scored on x only.



## Group C: held-out faults (defined now, before running)



None of these are in the system prompt by name. The two ghost scenarios built during

development (`ghost_everywhere`, `ghost_in_range`) are **not** used here; group C uses
other parameters and seeds. Hot pixel and drift are implemented after this commit,
exactly as specified, with tests, and without changing the agent's prompt for them.

| Case | Base | Seed | Fault (exact definition) | Expected verdict |
|---|---|---|---|---|
| C1 | clean | 300 | Ghost everywhere: `Ghost(relative_power=0.08, offset_x_m=-0.35e-3, offset_y_m=0.25e-3)` | measurement_failed, with the ghost named as the cause |

| C2 | clean | 301 | Ghost only for 170 ≤ z ≤ 250 mm: `Ghost(relative_power=0.12, offset_x_m=0.4e-3, offset_y_m=0.0, z_min_m=0.170, z_max_m=0.250)` | iso_compliant (ghost positions rejected, far points taken below the waist) |

| C3 | clean | 302 | Hot pixel cluster appearing after the dark frames: 2 × 2 pixels at rows 40-41, columns 470-471, +2500 counts in every light frame (not in darks) | iso_compliant, M² within 3 % (affected frames rejected or the defect explained) |
| C4 | clean | 303 | Slow pointing drift: beam centre x0 = +25 µm × (number of measurements taken so far), y0 = 0 | iso_compliant (drift noticed and named) |



## Metrics



Per run:

- **Correct verdict**: equals the expected verdict (group B3: x only).

- **M² error**: (M²_fit / M²_true − 1) per axis, when M² is determined; reported, and

  summarised as the median and maximum of |error| over cases with a determined M².

- **Constraint violations** (group B): count per the table, from the tool log.

- **Positions used**: measurements plus dark retakes.

- **Cost**: model calls, input/cached/output tokens, USD (prices at commit time;

  check the pricing page).

- **Fault noticed** (group C, agent): the fault is named in a `reason` or in the

  explanation. Keywords decided now: ghost → "ghost", "second spot", "reflection",

  "double"; hot pixel → "hot pixel", "bright pixel", "stuck", "defect"; drift →

  "drift", "moving", "centroid".



## Budget and stopping



Estimated agent cost: about $0.08 per run, 31 agent runs → about $2.50. Hard stop if

the agent runs of this evaluation pass **$4** in total; the remaining cases are then

reported as not run.



## Reporting



One table per group with all contenders, the metrics above, and a short note for every

wrong verdict explaining what went wrong (from the reasons log). Saved runs of the

agent go to `runs/evaluation/` (not committed); a summary table goes to

`docs/evaluation-results.md`.



## Known limitations of this plan



- One run per case: agent results vary from run to run; differences of one or two

  cases are not significant.

- The scripted baseline was developed against the six known scenarios and the

  developer saw the mystery-beam generator; group A favours it, by design.

- Group B is not a contest the scripted baseline can win; it shows what notes change.

- The oracle rule is a simple geometric judgement, not the standard's text.



## Deviations



All of these are dated 27 Oct 2026 and committed **before any evaluation run**.



1. **C1 is no longer held out.** The development scenario `ghost_everywhere` is the
   same fault type, and after seeing the agent fail on it (it never looked at a frame)
   the system prompt was changed (deviation 6). C1 is still run with its seed and
   definition, but it is reported separately as **"seen"** and left out of the
   held-out totals. C2 is the same fault type as the development scenario
   `ghost_in_range`; that was known when the plan was written, and the prompt was
   not changed after that run, so C2 stays held out as planned.
2. **C1 wording fix.** At 8 % power the ghost stays outside the iterated 3·d
   integration area near the waist, so the waist widths come out clean and only the
   far field is inflated. The sentence "no position gives true widths" was wrong for
   C1. The expected verdict (`measurement_failed`, ghost named) is unchanged,
   because no ISO-compliant far field can be measured on either side.
3. **New held-out case C5: an iris that clips the beam in a z range.**

   | Case | Base | Seed | Fault (exact definition) | Expected verdict |
   |---|---|---|---|---|
   | C5 | clean | 304 | Circular aperture in front of the sensor for 30 ≤ z ≤ 70 mm: `Aperture(radius_m=0.35e-3, z_min_m=0.030, z_max_m=0.070)`, centred on the beam axis; the profile is truncated, diffraction ignored | iso_compliant, M² within 3 % (clipped positions rejected or avoided) |



   Widths inside the range come out too small (−21 % at 30 mm, −11 % at 50 mm,

   about −1.5 to −2.5 % at 70 mm depending on exposure and noise). This was checked once on the simulator, without the agent, to

   make sure that **no current flag fires**. Far points are available outside the

   iris range below the waist (below 30 mm) and above it (170–250 mm). Fault-noticed

   keywords: "aperture", "iris", "truncat", "vignett", "cut off", "obstruct",

   "smaller than expected".

4. **Ablation: group C runs twice for the agent.** `agent` is the normal agent with

   `view_frame` (4 views). `agent-noview` has the tool removed from its tool list,
   the "Looking at frames" section removed from its system prompt, and a view
   budget of 0. Same seeds and cases. The scripted and naive contenders run group C
   once (they never look at frames).
5. **Scoring of C3 and C5.** Their expected column includes "M² within 3 %". A run
   counts as correct only if the verdict matches **and** |M² error| ≤ 3 % on both
   axes.
6. **Code changes after pre-registration, before any run.**
   - The system prompt now names general triggers for looking at a frame: a flag at
     every position, an azimuth that stays tilted, widths that do not follow a

     smooth hyperbola. It does not name ghosts, apertures or any group C fault.

   - Only the 2 most recent images stay in the resent history; older ones become a

     short text stub.

   - Hot pixels, pointing drift and the aperture were added to the simulator.

7. **Budget and order.** Agent runs: 16 (A) + 5 (B) + 2 × 5 (C) = 31. The original

   "31" was a miscount of 25. The stop rule stays at **$4** in total for this

   evaluation's agent runs. Order: A1–A16, B1–B5, then C1–C5, each case with and

   without `view_frame`. Up to 3 runs go in parallel. The budget is checked before

   each run starts, so at most the runs already going (≤ 2 × $0.50 per-run cap)

   can pass $4.

8. **Scoring details and oracle output, fixed in `scripts/run_evaluation.py`

   before any agent run.**

   - Expected verdicts of the mystery beams from the oracle rule: A7–A10

     `iso_compliant`, A11 `waist_not_accessible`, A12–A16 `iso_compliant`. As a

     check, the rule gives the same verdicts as the table for A1–A6.

   - B3 (x only): the `finish` guard does not allow `iso_compliant` while y

     coverage is missing, so B3 is scored on the verdict the x axis alone supports.

     It is `iso_compliant` if the final evaluation has x coverage met, x M²

     determined, no M² < 1 flag on x and no general astigmatism;

     `estimate_not_compliant` if x M² is determined; otherwise

     `measurement_failed`.

   - Violations are counted on `measure_at` calls that produced a measurement. B2

     counts 1 if more than 12 positions were used.

   - Fault noticed (agent only): the keywords are searched case-insensitively in

     all tool-call reasons and the `finish` explanation.

   - The scripted and naive contenders were run once, without any API call, while

     the script was being tested. They are deterministic, so the evaluation run

     reproduces those results. No agent run of an evaluation case happened before

     this commit.

9. **Caveats from the physics review, recorded before any agent run.**

   - **C2 and C5 are only partly held out.** No fault is named in the prompt, but

     the new look-triggers were written after seeing a ghost run. "Azimuth stays

     tilted" and "one side of the waist does not match the other" fit C2 (a ghost

     over 170–250 mm gives a tilted azimuth and a lopsided caustic) and C5. C2 and

     C5 stay in the held-out totals as planned, and the report states this caveat

     next to them.

   - **C3 mainly breaks the exposure feedback, not the widths.** Whenever the beam

     peak is below the hot cluster (2,500 counts), the frame maximum is the cluster.

     `peak_counts` then reads about 2,500 whatever the exposure, `low_signal` never

     fires, and at low exposure the width iteration locks onto the cluster

     (`too_few_pixels` or `not_converged`). With a beam peak above 2,500 the widths

     are biased by at most +0.3 %. The expected verdict is achievable, and the

     baselines fail it (deviation 8). The agent may describe it as "beam too small"

     without a fault keyword, so "fault noticed" for C3 is also checked by hand

     against the reasons log.

   - **The C4 keyword "centroid" is weak.** The agent can write "centroid" without

     noticing drift. "Fault noticed" for C4 is also checked by hand; both the

     keyword result and the hand check are reported.