Blame for docs/evaluation-plan-v3.md

# BeamPilot evaluation plan v3: inaccessible waist with change_lens



Status: pre-registered on 31 Oct 2026, **before any run on the v3 seeds 900–907**.

- **Code:** branch `add-change-lens` at `2a7dfd5` (not merged into main).

- **Runner:** `scripts/run_evaluation_v3.py`.

- **Cases and oracle:** `scripts/v3_cases.py`.
- **No code changes between this commit and the runs.** If something breaks, the
  evaluation stops and is reported; no case is re-run.
- **v1 and v2 stay unchanged.** Later changes go under "Deviations" at the end.

Design: [change-lens-design.md](change-lens-design.md). Progress, margin table and
open issues: [change-lens-progress.md](change-lens-progress.md).

## Question

When the waist lies outside the stage, does the agent use a thin lens to form an
artificial waist, measure it to ISO 11146-1, and give back the original beam (d0,
z0) and M² within tolerance? And does it say `waist_not_accessible` honestly when
no kit lens can help? It is compared with a fair scripted operator that has a
fixed lens rule.

## Contenders

| Name | What it is |
|---|---|
| `agent` | `run_agent`, `muse-spark-1.2`, effort `low`, limits 60 model calls, 1.5 M tokens, $0.50 per run; `AgentConfig(change_lens=True)`, `view_frame` on |
| `scripted` | `scripted_iso` with the lens rule (design section 6): first listed lens option, then scouts from the prediction (at least 4 positions) and its normal fill |
| `naive` | `naive_uniform`, reference only (never uses a lens) |

Every case has the lens kit (+25, +40, +300, +750, −200 mm; rail −300…−20 mm in
10 mm steps) and **28 positions**, for every contender. One run per case.

## Cases (seeds fixed now)

| Case | Seed | Waist z0 (mm) | d0 (µm) | M² | Expected (oracle) | Lens margin |
|---|---|---|---|---|---|---|
| L1 | 900 | −30 | 200 | 1.2 | iso_compliant | **borderline** (1 lens; best works at 12/27 corners) |

| L2 | 901 | −60 | 200 | 1.2 | iso_compliant | ok (2 lenses; 27/27) |
| L3 | 902 | −100 | 250 | 1.4 | iso_compliant | ok (5; 27/27) |

| L4 | 903 | −80 | 160 | 1.0 | iso_compliant | **borderline** (6; best 23/27) |
| L5 | 904 | +280 | 200 | 1.2 | iso_compliant | ok (29; 27/27) |

| L6 | 905 | +320 | 300 | 1.6 | iso_compliant | ok (46; 27/27) |
| L7 | 906 | +380 | 250 | 1.4 | iso_compliant | ok (29; 27/27) |

| L8 | 907 | +260 | 130 | 1.0 | waist_not_accessible | no lens at any of the 27 corners |



Both axes have the same d0, z0 and M². Wavelength 1064 nm; the camera and stage are

the same as in v1 and v2.



**How each column was set:**

- **Expected verdict:** the oracle below, applied to the true beam.

- **Lens margin:** the best feasible lens for the true beam, checked at the 27

  corners z0 ±10 mm, d0 ±10 %, M² ±10 %.

- **L8:** chosen by a recorded search (`choose_l8`):
  - grid: waist −150…−10 mm and 260…500 mm, d0 100…350 µm, M² 1.0…1.8;
  - at least 10 usable stage positions;
  - no feasible lens for the true beam or for any of the 27 corners;
  - the most usable positions wins (25); 87 qualifying candidates.

**Oracle with lenses** (`scripts/v3_cases.py`):
1. `measurement_failed` if fewer than 5 stage positions (5 mm grid) give a usable
   beam size: at least 20 px across and at most half the smaller sensor side.
2. Waist inside the stage: the v1 rule (room for 5 near and 5 far points per
   axis).
3. Waist outside: `iso_compliant` if any kit × rail lens gives an artificial
   waist that passes that rule on both axes (inside the stage, ≥ 20 px, beam
   ≤ 8 mm at the lens); otherwise `waist_not_accessible`.

## Metrics

- **Correct verdict:** equals the oracle's expected verdict.
- **Within tolerance** (cases expected `iso_compliant`), on both axes:
  - |M² error| ≤ 3 %;
  - back-calculated original d0 within 5 %;
  - back-calculated z0 within 0.1 × the true zR.
- **Reported, not scored:**
  - the lens chosen and whether it was a listed option;
  - lens changes;
  - positions;
  - model calls, views and cost;
  - the "dz0 per +1 % f" sensitivity.

## Calibration and development runs (none on 900–907)

**Scripted calibration, seeds 950–957** (L1 → 950 … L8 → 957), 28 positions:

| Case | Lens | Positions | Verdict | Worst error d0 / z0 / M² | dz0 per +1 % f |
|---|---|---|---|---|---|
| L1 | −200 @ −110 | 24 | iso_compliant | 0.1 % / 0.004 zR / 0.1 % | +0.29 mm |

| L2 | +25 @ −20 | 19 | iso_compliant | 0.2 % / 0.002 zR / 0.3 % | −0.40 mm |
| L3 | +40 @ −40 | 19 | iso_compliant | 0.3 % / 0.000 zR / 0.5 % | −0.62 mm |

| L4 | +40 @ −20 | 19 | iso_compliant | 0.1 % / 0.003 zR / 0.5 % | −0.81 mm |
| L5 | +750 @ −20 | 26 | iso_compliant | 0.2 % / 0.005 zR / 0.2 % | −1.18 mm |

| L6 | +750 @ −20 | 24 | iso_compliant | 0.3 % / 0.001 zR / 0.4 % | −1.50 mm |
| L7 | +750 @ −50 | 23 | iso_compliant | 0.2 % / 0.002 zR / 0.5 % | −2.41 mm |

| L8 | none | 13 | waist_not_accessible | – | – |



- **Spread over the 7 lens cases:** d0 ≤ 0.3 %, z0 ≤ 0.005 zR, M² ≤ 0.5 %, well

  inside the tolerances.

- **Development dry run (seeds 60–75, two per case):** 16 of 16 runs reached the

  expected verdict, with worst errors d0 0.6 %, z0 0.012 zR, M² 1.0 %.



**One live agent smoke run** (`waist_outside_range`, waist −50 mm, seed 920, 28

positions; not a v3 case):

- **Outcome:** `iso_compliant`, method `artificial_waist`.

- **Lens:** +25 mm at −20 mm, the listed option with fraction 1.0.

- **Accuracy:** back-calculated d0 +0.2 %, z0 +0.002 zR, M² +0.3 % on both axes.

- **Effort and cost:** 41 model calls, 21 positions, $0.133.



## Budget and stopping



- **Estimate:** 8 agent runs at about $0.10–0.15 → about $1.

- **Hard stop at $2** for the agent runs; the remaining cases are then reported as

  not run.

- **Order:** scripted and naive for all cases first (no API), then the agent L1 to

  L8, one at a time.



## Known limitations



- **Small N.** One run per case, so a difference of one or two cases is noise.

- **Not held out.** The cases, the kit, the scripted lens rule and the agent's lens

  prompt were developed together, with dry runs on development seeds. v3 measures

  whether the feature works end to end, not generalisation to unseen faults.

- **Fixes during development.** After the first dry run L8 was re-selected twice

  and the scripted scout rule was fixed; no fix was aimed at a single case (see

  the progress note).

- **Borderline cases.** L1 and L4 are borderline: their best lens fails at some of

  the ±10 % corners. L8 lies 10 mm past the stage end (about 0.8 zR).

- **Thin-lens model, exact f.** Real lenses (±1 % focal length) would add the

  listed "per +1 % f" error to z0. The wording stays "ISO 11146-1 procedure

  applied at an artificial waist (thin-lens model)", never "ISO compliant for the

  original beam" (design section 10).



## Deviations



1. **31 Oct 2026, before any v3 run: higher agent limits and budget.**

   - **Reason:** the smoke run used 41 of 60 model calls for 21 positions, and

     L5–L7 need 23–26 positions, so the 60-call limit would probably decide some

     runs (as A8 and C2 in v1).

   - **For v3 only:** the agent limits are **80 model calls** and **3 M tokens**;

     the per-run cost cap stays at **$0.50**.

   - **The total hard stop rises from $2 to $3.**

   - **Change:** only the settings in `scripts/run_evaluation_v3.py`

     (`AGENT_CONFIG`, `BUDGET_USD`); no other code changed. Commit `58e1b29`.

   - **The v3 runs use the code at `58e1b29`.**