docs/agent-design.md

BeamPilot agent design (v1)

Status: v1, agreed 26 Oct 2026. Updated 27 Oct 2026 (added retake_darks). All data is simulated. No lab data, no code from other projects.

1. Goal and pitch

BeamPilot is an agent that measures the M² of a laser beam to ISO 11146 with a camera on a moving stage (simulated). It decides where to measure, judges whether each measurement can be trusted, recovers from faults, and gives an honest final verdict, including "this cannot be measured to the standard with this setup".

Why an LLM and not a fixed script: commercial M² instruments already place ISO points automatically on clean beams. BeamPilot's value is judgement when things go wrong (saturation, background drift, beam too large for the sensor, waist out of reach, different x/y waists) and explaining every decision. We test this claim against a smart scripted baseline and report the result honestly, even if the agent only wins on fault scenarios.

2. Who does what

Part	Job	Never does
Muse Spark (agent)	Chooses positions and exposure, accepts or rejects measurements, decides when to finish, explains each action	Compute or type physics numbers. The report numbers come from Python state.
Python tools	Simulator, ISO 11146 widths, weighted caustic fit, coverage and sanity checks	Decide what to measure next or discard data on their own
Python controller	Runs the loop, holds all state, enforces hard limits, logs everything	Trust the agent to stop or to report correct numbers

The phases in section 5 are guidance in the system prompt, not controller code. The model chooses the order of actions; the controller only enforces limits.

3. Units

• Python computes internally in SI (metres, radians).
• The tool interface shows z in mm, diameters in µm and wavelength in nm, because that is easier for the model to read. Conversion happens only at the tool boundary.
• A known-answer test checks M² end to end, so a unit error (for example a factor 10³) fails.

4. Tools (7 core)

Every tool has a required reason string. Tool schemas are strict, and verdicts are enums.

1. get_setup(reason)
   • Wavelength, stage range, pixel size, sensor size, exposure limits, position budget left.
2. measure_at(z_mm, exposure_ms, reason)
   • Moves the stage, takes 5 repeated frames (ISO 11146-1 asks for at least 5 per position), applies background correction, computes ISO second-moment diameters.
   • Returns: measurement_id, mean ± std of d_x and d_y, quality flags (saturation fraction, peak / full scale, edge clipping, pixels per beam width, residual background, ellipse azimuth), and a suggested verdict. The measurement starts as pending.
   • Several measure_at calls may come in one turn (parallel tool calls); the controller runs them one after another, like a real stage.
3. set_status(measurement_id, status, reason)
   • status is accept or reject. Only accepted measurements go into the fit.
   • Guardrail: the controller refuses to accept a saturated measurement.
4. evaluate(reason)
   • Weighted caustic fit per axis, per-axis ISO coverage in units of zR ("x: 3 within ±1 zR, need about 5; 1 beyond 2 zR, need about 5"), waist accessibility (inside / outside / uncertain), general-astigmatism check, M² < 1 sanity flag, and a compact table of all measurements.
   • If an axis fit fails, or no accepted position lies within ±1 zR of its fitted waist, d0, z0, zR and M² of that axis are reported as null with a not_determined reason (and waist_side: below/above the measured positions or within a gap, waist_gap_mm), never as an extrapolated number. The same applies to the finish report.
   • Waist accessibility per axis from z0 ± 2σ against the stage range: inside, outside or uncertain (then waist_near_stage_end: measure closer to that end). Only outside supports the waist_not_accessible verdict.
5. finish(verdict, explanation)
   • verdict is one of: iso_compliant, estimate_not_compliant, waist_not_accessible, general_astigmatism_out_of_scope, measurement_failed.
   • Python builds the report numbers from stored state. The model writes only the verdict and the text.
   • Guardrail: if the model claims iso_compliant but evaluate says coverage is not met, or flags general astigmatism or M² < 1, the controller returns an error and the agent must continue or choose another verdict. waist_not_accessible is refused unless an axis is outside and an accepted position lies within 5 % of the stage length from that stage end. A waist up to about 1 zR outside the stage may still give an accurate M²; the verdict is then waist_not_accessible by definition (no near-waist points on both sides), not because the number is unusable.
   • If a hard limit (model calls, tokens, cost) or an API error stops the agent first, Python writes a fallback report from session state: estimate_not_compliant if M² is determined on both axes, otherwise measurement_failed, with stopped_by_limit holding the reason.
6. retake_darks(reason)
   • Blocks the beam and takes a fresh dark set (50 frames) with the current background, so a background change (e.g. room light switched on) becomes part of the dark.
   • Costs one position from the budget. Use it when measure_at reports background_changed.
7. view_frame(measurement_id, reason)
   • Optional. Returns the first frame of a measurement as a PNG image for the model (Muse Spark image input), on a square-root brightness scale, so the agent can see what the second moments hide (e.g. a ghost reflection).
   • Separate small budget (4 views per run), no position cost. The image goes to the model inside the tool output and is not stored in logs or run files.
   • Only the 2 most recent images stay in the resent history; older ones become a text stub. AgentConfig(view_frame=False) switches the tool off (evaluation ablation).

Stretch tool (only after the core works): change_lens(f_mm, reason) for an inaccessible waist (ISO uses an artificial waist made with an aberration-free lens, and M² stays the same).

5. Loop guidance (in the system prompt)

Scout:    get_setup, a few measurements spread over the stage range, fix exposure if needed
Locate:   evaluate → first estimates of z0 and zR for x and y
Fill:     measure where coverage is missing, accept/reject each result, evaluate again
Finish:   when coverage is met on both axes, or when the setup cannot meet it

Hard limits in the controller: maximum positions, maximum agent turns, token budget per run. Exact numbers are set after the first test runs.

6. Physics (ISO 11146-1)

• Second-moment diameter d = 4σ, per principal axis (x and y, not rotated).
• Integration area: centred on the centroid, 3 × the beam widths, iterated until it converges (2005 wording; the 2021 edition relaxed this, exact text unverified).
• Background: subtract a dark frame, then a fine offset from corner patches (2–5 % of the image), and keep negative pixel values (ISO/TR 11146-3).
• Caustic fit: d(z)² = a + b·z + c·z², fitted by minimising squared relative diameter deviations, weighted by the repeat variance (scipy.optimize.least_squares, not plain polyfit).
• Results: z0 = −b/(2c), d0 = √(a − b²/(4c)), Θ = √c (full angle), zR = d0/Θ, M² = (π/(8λ)) · √(4ac − b²).
• Point rule: at least 10 positions, about half within one zR of the waist and about half beyond 2 zR, applied separately for x and y.
• Detector rules: at least 20 pixels across the beam (spatial resolution ≥ 1/20 of the beam width), clipping loss under 1 % of the power. These create real trade-offs: far from the waist the beam can outgrow the sensor; near the waist it can get too small for the pixels.
• General astigmatism: if the azimuth of two non-circular profiles differs by more than 10°, ISO 11146-2 applies. That is out of scope: the agent detects it and stops.
• M² > 1 in the simulator: Gaussian profile with an enlarged width ("embedded Gaussian"). Hermite-Gaussian mode mixes are a later option.

Source of these points: the review read the ISO 11146-1:2005 and ISO/TR 11146-3 texts from third-party copies. The maintainer should check the key rules once before claiming compliance.

7. Scenarios (all simulated)

Core:

1. Clean beam, M² close to 1.
2. Saturation at some positions only.
3. Background drift during the run ("room light turned on").
4. Beam too large for the sensor at far positions.
5. Waist outside the stage range → waist_not_accessible with explanation.
6. Different waist positions in x and y (simple astigmatism). All code is per-axis from day one.

Ghost reflection (with view_frame): ghost_everywhere (expected measurement_failed) and ghost_in_range (expected iso_compliant after rejecting the ghost positions). Held-out faults for evaluation only (not in SCENARIOS): iris aperture over a z range, hot pixels, pointing drift (see docs/evaluation-plan.md).

Stretch: change_lens, general astigmatism detection demo.

Demo features: fault buttons the judge can press during a run, a "mystery beam" button with random hidden parameters (ground truth shown at the end), and a replay mode for recorded runs.

8. Evaluation

Three contenders on the same set of randomised hidden scenarios (fixed seed):

1. Naive uniform scan
2. Smart scripted ISO rule (scout, fit, place points, refit, simple fault rules)
3. Muse Spark agent

Metrics: ISO-compliance rate, M² error versus ground truth, faults noticed, positions used. The scripted baseline must be a fair, honest attempt, not built to fail. Batch size is decided after measuring the cost of one agent run.

9. Testing

• Known-answer tests: ideal Gaussian gives M² = 1; a known M² > 1 is recovered; constant offset is removed; near-waist-only sampling gives a biased result (divergence badly constrained).
• Optional test oracle: the open-source laserbeamsize package, as a dev dependency only (licence and use to be checked and agreed first).
• The scripted baseline also tests the tools and controller without spending API credits.

10. Muse Spark settings

• Responses API with function calling. Output cap is max_output_tokens (reasoning tokens count toward it). Resend the system instructions on every call (previous_response_id does not keep them). Tool results go back as function_call_output with the matching call_id.
• reasoning_effort low or medium for normal steps. Measure latency and tokens; don't guess.
• Development batches may use the cheaper contributor model (data used for training; we only send simulated data). Whether hackathon credits cover it: unverified.
• Log tokens and the reason field for every step.

11. Schedule (Nuremberg time; adjust to real available hours)

Day	Goal
Tue 27 Oct	Simulator (per-axis Gaussian, M², noise, offset, saturation, dark frame) + tests. Workshop 17:00.
Wed 28 Oct	ISO widths (corner offset, iterative 3× area) + tests. Workshop 17:00.
Thu 29 Oct	Weighted fit and evaluate (coverage, checks) + tests
Fri 30 Oct	Tool wrappers, controller, scripted baseline from the command line. Check HF Spaces plan.
Sat 31 Oct – Sun 1 Nov	Muse Spark loop, logging, first agent runs
Mon 2 Nov	Gradio UI (caustic plot, decision log, fault buttons, replay), deploy. Feature freeze.
Tue 3 Nov	Small honest evaluation, README, record the video
Wed 4 Nov	Buffer. Submit in the evening.

Drop in this order if behind: SAM 3.1 → ghost scenario and view_frame → change_lens → polished x/y-waist demo (keep the per-axis code). Never drop: known-answer tests, scripted baseline, replay mode, the video.

12. Open decisions

1. Hosting: Gradio on Hugging Face Spaces is the likely choice. Check the current free/paid rules for Spaces before deciding (the review reported changes; unverified).
2. laserbeamsize as a dev-only test oracle: yes or no.
3. Impact framing for the video: who benefits (for example small labs and teaching labs without a commercial M² instrument). To be written without claims we cannot back up.
4. Whether SAM 3.1 gets its one-hour test at all.