docs/evaluation-results.md

BeamPilot evaluation results

Run on 27 Oct 2026 exactly as pre-registered in evaluation-plan.md, including its Deviations. Code: main at 03fb2c1, model muse-spark-1.2, reasoning effort low, limits per run 40 model calls, 1.5 M tokens, $0.50. Script: scripts/run_evaluation.py --run, one run per case. No code changed between runs, nothing broke, no case was re-run, and the $4 stop rule was not reached.

Total agent cost: $2.56 ($2.05 for agent, $0.51 for agent-noview). The scripted and naive contenders make no API calls.

Pre-registration, all on 27 Oct 2026 and before any agent run of the evaluation (the first run file is from 20:51):

Commit	Time	What
d2f445e	19:03	Plan: groups, seeds, expected verdicts, metrics, stop rule
b3025e9	20:21	Deviations 1–7: C1 "seen", C1 wording, new C5, ablation, scoring, budget and order
9ca92df	20:33	Deviation 8: scoring details and oracle output, plus the evaluation script
cda0f73	20:37	Deviation 9: caveats from the physics review (C2–C5)

Read the results with these limits in mind:

• One run per case. Every case ran once, so a difference of one case between contenders (or between agent and agent-noview) is within run-to-run noise and is not evidence either way.
• C1 is "seen". The prompt was changed after the agent failed the same fault type in development (deviation 1), so C1 is kept out of the held-out totals.
• Group B is not a fair contest. The scripted baseline cannot read operator notes, by design. It runs each B case without its note and its violations are reported; that is a limit of fixed scripts, not a bug.
• The 40-call limit decided two cases. A8 and C2 were stopped by the pre-registered limit of 40 model calls before finish, and the controller wrote the fallback verdict. The cause and details are below.

Summary

Contender	A (16)	B (5)	C held out (C2–C5)	C1 (seen)	Mean positions A / B / C	Violations (B)	Cost
agent	15 / 16	4 / 5	2 / 4	1 / 1	14.4 / 13.8 / 14.6	0	$2.047
agent-noview	–	–	2 / 4	0 / 1	– / – / 17.2	–	$0.512
scripted	16 / 16	5 / 5	2 / 4	0 / 1	15.3 / 17.8 / 15.4	8	$0
naive	2 / 16	0 / 5	0 / 4	0 / 1	10 / 10 / 10	4	$0

• Group A (expected tie): roughly a tie. The agent got 15 of 16 against 16 of 16 for scripted, using slightly fewer positions (14.4 against 15.3 on average). Its one miss, A8, came from the model-call limit, not from a wrong judgement (see below). Agent |M² error| over all determined axes and groups: median 0.2 %, maximum 6.1 % (C5).
• Group B (notes): the agent kept every note (0 violations in 5 cases) and got 4 of 5 verdicts right. The scripted baseline cannot read notes: its 5 correct verdicts come with 8 violations (B1: 1 position in the blocked range; B2: 19 positions against a limit of 12; B4: 1 exposure above 20 ms; B5: 5 positions above 230 mm). As the plan says, this is not a fair contest; it shows what the notes change.
• Group C (held out, C2–C5): agent 2 / 4, scripted 2 / 4. Nobody solved C5 (iris) within the 3 % tolerance.
• Ablation (view_frame): 2 / 4 held-out cases with views and 2 / 4 without, but different cases (with: C3, C4; without: C2, C4). On the seen case C1, only the agent with views got the verdict right. With one run per case these one-case differences are noise, not a measured effect.

Every wrong verdict, with the reason

From the reasons logs in runs/evaluation/ (not committed).

agent (with view_frame)

• A8: it stopped at the 40-model-call limit with x at 4 of 5 near points; the controller's fallback report gave estimate_not_compliant. The agent made one tool call per model call (no batching). M² itself was right (+0.3 %).
• B2: it kept the 12-position note (0 violations) but spent 4 positions on scouts it had to repeat (saturated or low signal). With 8 usable positions, x had only 4 near and 4 far, so it finished honestly with estimate_not_compliant.
• C2: it looked at frames, found the double spot on the high side and rejected 180–230 mm ("bimodal double spot"). But it had accepted the 250 mm point earlier, which is inside the ghost range, and it hit the 40-call limit while filling the low side. Fallback verdict estimate_not_compliant; M² +0.9 / −0.6 %.
• C5: it never looked at a frame and took far points at 40 and 60 mm, inside the iris range. When those points came in, the fitted M² dropped from 1.19 to 1.12, but no trigger fired. It reported iso_compliant with M² −5.9 / −6.1 %, outside the 3 % tolerance.

agent-noview (ablation)

• C1 (seen): it accepted ghost-inflated widths at every position and read the constant axes_not_aligned (about 36°) as a rotated beam. It reported iso_compliant with M² +109 / +60 %. The finish guard allows this because coverage was met.
• C3: the hot pixel made peak_counts read about 2,500 at every exposure, and several scouts came back too_few_pixels or not_converged. It rejected 8 positions (m1 as a "corner artifact"), ran out of budget, and finished with estimate_not_compliant.
• C5: it saw the fit residual jump to about 6 % "at 30 mm" and said it would check, but kept the points. It reported iso_compliant with M² −7.4 / −7.3 %.

scripted

• C1 (seen): its rules accept the ghost-inflated widths, giving estimate_not_compliant with M² +109 / +60 %.
• C3: the hot pixel takes over peak_counts and the start area of the width iteration, so its scouts come back too_few_pixels. The rules give up after 4 positions with measurement_failed.
• C5: it accepted clipped far points and reported iso_compliant with M² −3.1 / −3.2 %, just outside the 3 % tolerance.
• Correct by the plan's scoring but worth noting: C2 iso_compliant with M² x +10.2 %. Ghost points were accepted, and C2 is scored on the verdict only.

naive (reference only): wrong in 24 of 26 cases.

• 10 evenly spaced positions at 1 ms cannot give 5 near and 5 far points per axis, so it reports estimate_not_compliant everywhere except in these cases:
  • A6: general_astigmatism_out_of_scope, flagged from its 10 points; the beam has none.
  • C3: measurement_failed (hot pixel).
  • A5 and A11: both correct, waist_not_accessible.

Also correct by the plan's scoring but worth noting: C2 agent-noview reported iso_compliant with M² x +6.7 % because it accepted ghost points.

Fault noticed (group C, agent)

Case	agent: keyword / by hand	agent-noview: keyword / by hand
C1 ghost (seen)	yes / yes ("secondary ghost beam")	no / no
C2 ghost	yes / yes ("bimodal double spot")	no / no
C3 hot pixel	no / no: viewed m3 ("anomalous tiny beam and off centroid") but never named a defect	no / partly ("corner artifact")
C4 drift	no / no	no / no
C5 iris	no / no	no / no: saw the residual jump, did not name a cause

The by-hand checks for C3 and C4 were pre-registered (deviation 9). In C4 the drift (up to about 0.3 mm) did not change the widths, so iso_compliant was right without noticing it. Views used per run with view_frame: C1 4, C2 4, C3 1, C4 0, C5 0. No view was used in groups A and B.

Observations not in the plan's metrics

• The model-call limit decided two cases. In 12 of 31 agent runs the model made exactly one tool call per model call instead of batching. A8 and C2 then hit the pre-registered 40-call limit before finish. All other runs finished on their own. Nothing was changed for this during the evaluation; raising the limit or nudging batching is a later decision.
• The look-triggers did not fire on C5. The iris changes no flag, and the "smooth hyperbola" trigger depends on the agent comparing widths, which it did not do. The ablation run noticed the residual jump but did not act on it.
• Caveats from the plan apply: C2 and C5 are only partly held out (deviation 9); C1 is reported as "seen" (deviation 1); one run per case.

Full tables

Generated by scripts/run_evaluation.py --summary from runs/evaluation/results.jsonl. M² err = M²_fit / M²_true − 1. Noticed = keyword check (agent only). Views = view_frame calls. B3 is scored on x only.

Group A

Case	Contender	Expected	Verdict	Correct	M² err x	M² err y	Violations	Positions	Noticed	Views	Cost
A1	agent	iso_compliant	iso_compliant	yes	+0.3 %	+0.2 %		14		0	$0.060
A1	naive	iso_compliant	estimate_not_compliant	no	+1.5 %	+1.1 %		10
A1	scripted	iso_compliant	iso_compliant	yes	+0.2 %	+0.0 %		19
A2	agent	iso_compliant	iso_compliant	yes	-0.2 %	-0.2 %		14		0	$0.100
A2	naive	iso_compliant	estimate_not_compliant	no	–	–		10
A2	scripted	iso_compliant	iso_compliant	yes	-0.1 %	-0.1 %		17
A3	agent	iso_compliant	iso_compliant	yes	-0.2 %	-0.2 %		16		0	$0.063
A3	naive	iso_compliant	estimate_not_compliant	no	+1.5 %	+1.1 %		10
A3	scripted	iso_compliant	iso_compliant	yes	+0.4 %	+0.2 %		20
A4	agent	iso_compliant	iso_compliant	yes	+0.3 %	+0.1 %		13		0	$0.085
A4	naive	iso_compliant	estimate_not_compliant	no	+7.7 %	+7.7 %		10
A4	scripted	iso_compliant	iso_compliant	yes	-0.0 %	-0.1 %		15
A5	agent	waist_not_accessible	waist_not_accessible	yes	–	–		14		0	$0.100
A5	naive	waist_not_accessible	waist_not_accessible	yes	–	–		10
A5	scripted	waist_not_accessible	waist_not_accessible	yes	–	–		8
A6	agent	iso_compliant	iso_compliant	yes	+0.2 %	+0.3 %		16		0	$0.065
A6	naive	iso_compliant	general_astigmatism_out_of_scope	no	+1.1 %	-0.6 %		10
A6	scripted	iso_compliant	iso_compliant	yes	-0.2 %	-0.3 %		17
A7	agent	iso_compliant	iso_compliant	yes	+0.0 %	+0.1 %		14		0	$0.060
A7	naive	iso_compliant	estimate_not_compliant	no	+17.2 %	+19.0 %		10
A7	scripted	iso_compliant	iso_compliant	yes	-0.7 %	-0.7 %		14
A8	agent	iso_compliant	estimate_not_compliant	no	+0.3 %	+0.3 %		18		0	$0.121
A8	naive	iso_compliant	estimate_not_compliant	no	-0.1 %	-0.2 %		10
A8	scripted	iso_compliant	iso_compliant	yes	+0.6 %	+0.6 %		16
A9	agent	iso_compliant	iso_compliant	yes	-0.8 %	-0.7 %		12		0	$0.080
A9	naive	iso_compliant	estimate_not_compliant	no	-0.2 %	-0.0 %		10
A9	scripted	iso_compliant	iso_compliant	yes	-0.2 %	-0.1 %		15
A10	agent	iso_compliant	iso_compliant	yes	+0.2 %	+0.2 %		16		0	$0.069
A10	naive	iso_compliant	estimate_not_compliant	no	+0.1 %	+0.1 %		10
A10	scripted	iso_compliant	iso_compliant	yes	-0.1 %	-0.2 %		16
A11	agent	waist_not_accessible	waist_not_accessible	yes	–	–		9		0	$0.055
A11	naive	waist_not_accessible	waist_not_accessible	yes	–	–		10
A11	scripted	waist_not_accessible	waist_not_accessible	yes	–	–		10
A12	agent	iso_compliant	iso_compliant	yes	+0.1 %	+0.5 %		15		0	$0.084
A12	naive	iso_compliant	estimate_not_compliant	no	-0.3 %	-0.1 %		10
A12	scripted	iso_compliant	iso_compliant	yes	+0.1 %	+0.2 %		20
A13	agent	iso_compliant	iso_compliant	yes	-0.0 %	-0.1 %		15		0	$0.093
A13	naive	iso_compliant	estimate_not_compliant	no	+2.3 %	+2.1 %		10
A13	scripted	iso_compliant	iso_compliant	yes	+0.5 %	+0.5 %		16
A14	agent	iso_compliant	iso_compliant	yes	+0.2 %	+0.0 %		16		0	$0.094
A14	naive	iso_compliant	estimate_not_compliant	no	+12.3 %	+11.4 %		10
A14	scripted	iso_compliant	iso_compliant	yes	+0.9 %	+0.8 %		16
A15	agent	iso_compliant	iso_compliant	yes	+0.4 %	+0.2 %		14		0	$0.058
A15	naive	iso_compliant	estimate_not_compliant	no	+9.8 %	+8.5 %		10
A15	scripted	iso_compliant	iso_compliant	yes	-0.1 %	-0.4 %		12
A16	agent	iso_compliant	iso_compliant	yes	+0.3 %	+0.1 %		14		0	$0.054
A16	naive	iso_compliant	estimate_not_compliant	no	+7.6 %	+5.9 %		10
A16	scripted	iso_compliant	iso_compliant	yes	+0.5 %	+0.2 %		14

Group B

Case	Contender	Expected	Verdict	Correct	M² err x	M² err y	Violations	Positions	Noticed	Views	Cost
B1	agent	iso_compliant	iso_compliant	yes	+0.5 %	+0.4 %	0	13		0	$0.054
B1	naive	iso_compliant	estimate_not_compliant	no	+2.0 %	+1.5 %	0	10
B1	scripted	iso_compliant	iso_compliant	yes	-0.0 %	-0.1 %	1	18
B2	agent	iso_compliant	estimate_not_compliant	no	-0.3 %	-0.3 %	0	12		0	$0.074
B2	naive	iso_compliant	estimate_not_compliant	no	-2.7 %	-2.8 %	0	10
B2	scripted	iso_compliant	iso_compliant	yes	-0.6 %	-0.6 %	1	19
B3	agent	iso_compliant	iso_compliant	yes	+0.3 %	–	0	17		0	$0.087
B3	naive	iso_compliant	estimate_not_compliant	no	+2.1 %	–	0	10
B3	scripted	iso_compliant	iso_compliant	yes	-0.3 %	–	0	18
B4	agent	iso_compliant	iso_compliant	yes	+0.0 %	+0.1 %	0	14		0	$0.067
B4	naive	iso_compliant	estimate_not_compliant	no	+2.4 %	+2.8 %	0	10
B4	scripted	iso_compliant	iso_compliant	yes	+0.1 %	+0.1 %	1	19
B5	agent	iso_compliant	iso_compliant	yes	+0.0 %	-0.0 %	0	13		0	$0.072
B5	naive	iso_compliant	estimate_not_compliant	no	+6.0 %	+6.2 %	4	10
B5	scripted	iso_compliant	iso_compliant	yes	+0.5 %	+0.5 %	5	15

Group C

Case	Contender	Expected	Verdict	Correct	M² err x	M² err y	Violations	Positions	Noticed	Views	Cost
C1 (seen)	agent	measurement_failed	measurement_failed	yes	–	–		16	yes	4	$0.119
C1 (seen)	agent-noview	measurement_failed	iso_compliant	no	+109.1 %	+59.6 %		17	no	0	$0.122
C1 (seen)	naive	measurement_failed	estimate_not_compliant	no	+104.3 %	+56.2 %		10
C1 (seen)	scripted	measurement_failed	estimate_not_compliant	no	+108.7 %	+59.9 %		17
C2	agent	iso_compliant	estimate_not_compliant	no	+0.9 %	-0.6 %		18	yes	4	$0.136
C2	agent-noview	iso_compliant	iso_compliant	yes	+6.7 %	-0.5 %		19	no	0	$0.103
C2	naive	iso_compliant	estimate_not_compliant	no	+3.0 %	-0.3 %		10
C2	scripted	iso_compliant	iso_compliant	yes	+10.2 %	+0.0 %		19
C3	agent	iso_compliant	iso_compliant	yes	+0.1 %	-0.0 %		13	no	1	$0.084
C3	agent-noview	iso_compliant	estimate_not_compliant	no	-0.2 %	-0.2 %		20	no	0	$0.099
C3	naive	iso_compliant	measurement_failed	no	–	–		10
C3	scripted	iso_compliant	measurement_failed	no	–	–		4
C4	agent	iso_compliant	iso_compliant	yes	-0.3 %	-0.3 %		13	no	0	$0.050
C4	agent-noview	iso_compliant	iso_compliant	yes	-0.2 %	-0.2 %		14	no	0	$0.093
C4	naive	iso_compliant	estimate_not_compliant	no	-0.5 %	-0.6 %		10
C4	scripted	iso_compliant	iso_compliant	yes	-0.6 %	-0.6 %		19
C5	agent	iso_compliant	iso_compliant	no	-5.9 %	-6.1 %		13	no	0	$0.060
C5	agent-noview	iso_compliant	iso_compliant	no	-7.4 %	-7.3 %		16	no	0	$0.095
C5	naive	iso_compliant	estimate_not_compliant	no	+3.2 %	+4.3 %		10
C5	scripted	iso_compliant	iso_compliant	no	-3.1 %	-3.2 %		18

Totals (correct / run)

Contender	A	B	C held out (C2-C5)	C1 (seen)	Cost
agent	15 / 16	4 / 5	2 / 4	1 / 1	$2.047
agent-noview	–	–	2 / 4	0 / 1	$0.512
scripted	16 / 16	5 / 5	2 / 4	0 / 1	$0.000
naive	2 / 16	0 / 5	0 / 4	0 / 1	$0.000

Agent |M² error| over determined axes: median 0.2 %, max 6.1 % (45 axes).