Blame for docs/agent-design.md

# BeamPilot agent design (v1)

Status: v1, agreed 26 Oct 2026.
Updated 27 Oct 2026 (added `retake_darks`). All data is simulated. No lab data, no code from other projects.

## 1. Goal and pitch

BeamPilot is an agent that measures the M² of a laser beam to ISO 11146 with a camera
on a moving stage (simulated). It decides where to measure, judges whether each
measurement can be trusted, recovers from faults, and gives an honest final verdict,
including "this cannot be measured to the standard with this setup".

Why an LLM and not a fixed script: commercial M² instruments already place ISO points
automatically on clean beams. BeamPilot's value is **judgement when things go wrong**
(saturation, background drift, beam too large for the sensor, waist out of reach,
different x/y waists) and **explaining every decision**. We test this claim against a
smart scripted baseline and report the result honestly, even if the agent only wins on
fault scenarios.

## 2. Who does what

| Part | Job | Never does |
|---|---|---|
| Muse Spark (agent) | Chooses positions and exposure, accepts or rejects measurements, decides when to finish, explains each action | Compute or type physics numbers. The report numbers come from Python state. |
| Python tools | Simulator, ISO 11146 widths, weighted caustic fit, coverage and sanity checks | Decide what to measure next or discard data on their own |
| Python controller | Runs the loop, holds all state, enforces hard limits, logs everything | Trust the agent to stop or to report correct numbers |

The phases in section 5 are guidance in the system prompt, not controller code.
The model chooses the order of actions; the controller only enforces limits.

## 3. Units

- Python computes **internally in SI** (metres, radians).
- The tool interface shows z in mm, diameters in µm and wavelength in nm, because that is
  easier for the model to read. Conversion happens only at the tool boundary.
- A known-answer test checks M² end to end, so a unit error (for example a factor 10³) fails.

## 4. Tools (7 core)

Every tool has a required `reason` string. Tool schemas are strict, and verdicts are enums.

1. `get_setup(reason)`
   - Wavelength, stage range, pixel size, sensor size, exposure limits, position budget left.
2. `measure_at(z_mm, exposure_ms, reason)`
   - Moves the stage, takes **5 repeated frames** (ISO 11146-1 asks for at least 5 per position),
     applies background correction, computes ISO second-moment diameters.

   - Returns: `measurement_id`, mean ± std of d_x and d_y, quality flags (saturation fraction,

     peak / full scale, edge clipping, pixels per beam width, residual background, ellipse azimuth),

     and a *suggested* verdict. The measurement starts as `pending`.

   - Several `measure_at` calls may come in one turn (parallel tool calls); the controller runs

     them one after another, like a real stage.

3. `set_status(measurement_id, status, reason)`

   - `status` is `accept` or `reject`. Only accepted measurements go into the fit.

   - Guardrail: the controller refuses to accept a saturated measurement.

4. `evaluate(reason)`

   - Weighted caustic fit **per axis**, per-axis ISO coverage in units of zR

     ("x: 3 within ±1 zR, need about 5; 1 beyond 2 zR, need about 5"),

     waist accessibility (inside / outside / uncertain), general-astigmatism check, M² < 1

     sanity flag,

     and a compact table of all measurements.

   - If an axis fit fails, or no accepted position lies within ±1 zR of its fitted waist,

     d0, z0, zR and M² of that axis are reported as null with a `not_determined` reason

     (and `waist_side`: below/above the measured positions or within a gap,

     `waist_gap_mm`), never as an extrapolated number. The same applies to the `finish`

     report.

   - Waist accessibility per axis from z0 ± 2σ against the stage range: `inside`,

     `outside` or `uncertain` (then `waist_near_stage_end`: measure closer to that end).

     Only `outside` supports the `waist_not_accessible` verdict.

5. `finish(verdict, explanation)`

   - `verdict` is one of: `iso_compliant`, `estimate_not_compliant`, `waist_not_accessible`,

     `general_astigmatism_out_of_scope`, `measurement_failed`.

   - Python builds the report numbers from stored state. The model writes only the verdict and the text.

   - Guardrail: if the model claims `iso_compliant` but `evaluate` says coverage is not met,

     or flags general astigmatism or M² < 1, the controller returns an error and the agent

     must continue or choose another verdict. `waist_not_accessible` is refused unless an

     axis is `outside` and an accepted position lies within 5 % of the stage length from

     that stage end. A waist up to about 1 zR outside the stage may still give an accurate

     M²; the verdict is then `waist_not_accessible` by definition (no near-waist points on

     both sides), not because the number is unusable.

   - If a hard limit (model calls, tokens, cost) or an API error stops the agent first,

     Python writes a fallback report from session state: `estimate_not_compliant` if M²

     is determined on both axes, otherwise `measurement_failed`, with `stopped_by_limit`

     holding the reason.

6. `retake_darks(reason)`

   - Blocks the beam and takes a fresh dark set (50 frames) with the current background, so a

     background change (e.g. room light switched on) becomes part of the dark.

   - Costs one position from the budget. Use it when `measure_at` reports `background_changed`.

7. `view_frame(measurement_id, reason)`

   - Optional. Returns the first frame of a measurement as a PNG image for the model (Muse

     Spark image input), on a square-root brightness scale, so the agent can see what the

     second moments hide (e.g. a ghost reflection).

   - Separate small budget (4 views per run), no position cost. The image goes to the model

     inside the tool output and is not stored in logs or run files.

   - Only the 2 most recent images stay in the resent history; older ones become a text

     stub. `AgentConfig(view_frame=False)` switches the tool off (evaluation ablation).


Stretch tool (only after the core works): `change_lens(f_mm, reason)` for an inaccessible
waist (ISO uses an artificial waist made with an aberration-free lens, and M² stays the same).

## 5. Loop guidance (in the system prompt)

```

Scout:    get_setup, a few measurements spread over the stage range, fix exposure if needed

Locate:   evaluate → first estimates of z0 and zR for x and y

Fill:     measure where coverage is missing, accept/reject each result, evaluate again

Finish:   when coverage is met on both axes, or when the setup cannot meet it

```

Hard limits in the controller: maximum positions, maximum agent turns, token budget per run.
Exact numbers are set after the first test runs.

## 6. Physics (ISO 11146-1)

- Second-moment diameter d = 4σ, per principal axis (x and y, not rotated).
- Integration area: centred on the centroid, 3 × the beam widths, iterated until it converges
  (2005 wording; the 2021 edition relaxed this, exact text unverified).
- Background: subtract a dark frame, then a fine offset from corner patches (2–5 % of the image),
  and **keep negative pixel values** (ISO/TR 11146-3).
- Caustic fit: d(z)² = a + b·z + c·z², fitted by minimising squared **relative** diameter
  deviations, weighted by the repeat variance (`scipy.optimize.least_squares`, not plain `polyfit`).
- Results: z0 = −b/(2c), d0 = √(a − b²/(4c)), Θ = √c (full angle), zR = d0/Θ,
  M² = (π/(8λ)) · √(4ac − b²).
- Point rule: at least 10 positions, about half within one zR of the waist and about half beyond
  2 zR, applied **separately for x and y**.
- Detector rules: at least 20 pixels across the beam (spatial resolution ≥ 1/20 of the beam width),
  clipping loss under 1 % of the power. These create real trade-offs: far from the waist the beam
  can outgrow the sensor; near the waist it can get too small for the pixels.
- General astigmatism: if the azimuth of two non-circular profiles differs by more than 10°,
  ISO 11146-2 applies. That is out of scope: the agent detects it and stops.
- M² > 1 in the simulator: Gaussian profile with an enlarged width ("embedded Gaussian").
  Hermite-Gaussian mode mixes are a later option.

Source of these points: the review read the ISO 11146-1:2005 and ISO/TR 11146-3 texts from
third-party copies. The maintainer should check the key rules once before claiming compliance.

## 7. Scenarios (all simulated)

Core:
1. Clean beam, M² close to 1.
2. Saturation at some positions only.
3. Background drift during the run ("room light turned on").
4. Beam too large for the sensor at far positions.
5. Waist outside the stage range → `waist_not_accessible` with explanation.
6. Different waist positions in x and y (simple astigmatism). All code is per-axis from day one.

Ghost reflection (with `view_frame`): `ghost_everywhere` (expected `measurement_failed`) and
`ghost_in_range` (expected `iso_compliant` after rejecting the ghost positions).
Held-out faults for evaluation only (not in `SCENARIOS`): iris aperture over a z range,
hot pixels, pointing drift (see `docs/evaluation-plan.md`).

Stretch: `change_lens`, general astigmatism detection demo.

Demo features: fault buttons the judge can press during a run, a "mystery beam" button with
random hidden parameters (ground truth shown at the end), and a replay mode for recorded runs.

## 8. Evaluation

Three contenders on the same set of randomised hidden scenarios (fixed seed):
1. Naive uniform scan
2. Smart scripted ISO rule (scout, fit, place points, refit, simple fault rules)
3. Muse Spark agent

Metrics: ISO-compliance rate, M² error versus ground truth, faults noticed, positions used.
The scripted baseline must be a fair, honest attempt, not built to fail.
Batch size is decided after measuring the cost of one agent run.

## 9. Testing

- Known-answer tests: ideal Gaussian gives M² = 1; a known M² > 1 is recovered; constant offset
  is removed; near-waist-only sampling gives a biased result (divergence badly constrained).
- Optional test oracle: the open-source `laserbeamsize` package, as a dev dependency only
  (licence and use to be checked and agreed first).
- The scripted baseline also tests the tools and controller without spending API credits.

## 10. Muse Spark settings

- Responses API with function calling. Output cap is `max_output_tokens` (reasoning tokens count
  toward it). Resend the system instructions on every call (`previous_response_id` does not keep them).
  Tool results go back as `function_call_output` with the matching `call_id`.
- `reasoning_effort` low or medium for normal steps. Measure latency and tokens; don't guess.
- Development batches may use the cheaper contributor model (data used for training; we only
  send simulated data). Whether hackathon credits cover it: unverified.
- Log tokens and the `reason` field for every step.

## 11. Schedule (Nuremberg time; adjust to real available hours)

| Day | Goal |
|---|---|
| Tue 27 Oct | Simulator (per-axis Gaussian, M², noise, offset, saturation, dark frame) + tests. Workshop 17:00. |
| Wed 28 Oct | ISO widths (corner offset, iterative 3× area) + tests. Workshop 17:00. |
| Thu 29 Oct | Weighted fit and `evaluate` (coverage, checks) + tests |
| Fri 30 Oct | Tool wrappers, controller, scripted baseline from the command line. Check HF Spaces plan. |
| Sat 31 Oct – Sun 1 Nov | Muse Spark loop, logging, first agent runs |
| Mon 2 Nov | Gradio UI (caustic plot, decision log, fault buttons, replay), deploy. **Feature freeze.** |
| Tue 3 Nov | Small honest evaluation, README, record the video |
| Wed 4 Nov | Buffer. Submit in the evening. |

Drop in this order if behind: SAM 3.1 → ghost scenario and `view_frame` → `change_lens` →
polished x/y-waist demo (keep the per-axis code). Never drop: known-answer tests, scripted
baseline, replay mode, the video.

## 12. Open decisions

1. Hosting: Gradio on Hugging Face Spaces is the likely choice. Check the current free/paid
   rules for Spaces before deciding (the review reported changes; unverified).
2. `laserbeamsize` as a dev-only test oracle: yes or no.
3. Impact framing for the video: who benefits (for example small labs and teaching labs without a
   commercial M² instrument). To be written without claims we cannot back up.
4. Whether SAM 3.1 gets its one-hour test at all.