A newer version of the Gradio SDK is available: 6.31.0
BeamPilot evaluation summary (v1 + v2)
Generated by scripts/make_evaluation_summary.py from docs/results/evaluation-v1.jsonl and docs/results/evaluation-v2.jsonl. No number on this page is typed by hand.
Plans
- v1: evaluation-plan.md, pre-registered in
d2f445e(plan),b3025e9(deviations 1-7),9ca92df(deviation 8 and evaluation script),cda0f73(deviation 9); run onmainat03fb2c1. Full results: evaluation-results.md. - v2: evaluation-plan-v2.md, pre-registered in
75fe382(plan),3241612(deviations 1-2); run onmainat3403a76. Full results: evaluation-results-v2.md.
Between v1 and v2 only two things changed: the model-call limit (40 → 60) and a prompt asking for parallel tool calls. Seeds differ.
Read this first
- Small N. Each case ran twice in total: once in v1 and once in v2, with different seeds. A difference of one or two cases is noise, not a measured effect.
- C1 is "seen". The prompt was changed after the agent failed the same fault type during development, so C1 is never counted as held out. C2-C5 were held out in v1 and seen in v2.
- The scripted baseline cannot read operator notes, by design. In group B it runs without the note; its violations are counted below. Group B is therefore not a fair contest.
- The naive scan is a reference only (10 fixed positions, 1 ms).
Correct verdicts
Correct / run, per version and both together. Group C is run twice per version for the agent: with view_frame (agent) and without (agent-noview).
| Contender | A: v1 · v2 · both | B: v1 · v2 · both | C2-C5: v1 · v2 · both | C1 (seen): v1 · v2 · both |
|---|---|---|---|---|
| agent | 15/16 · 16/16 · 31/32 | 4/5 · 4/5 · 8/10 | 2/4 · 3/4 · 5/8 | 1/1 · 0/1 · 1/2 |
| agent-noview | – | – | 2/4 · 3/4 · 5/8 | 0/1 · 0/1 · 0/2 |
| scripted | 16/16 · 14/16 · 30/32 | 5/5 · 5/5 · 10/10 | 2/4 · 3/4 · 5/8 | 0/1 · 0/1 · 0/2 |
| naive | 2/16 · 1/16 · 3/32 | 0/5 · 0/5 · 0/10 | 0/4 · 0/4 · 0/8 | 0/1 · 0/1 · 0/2 |
Effort, accuracy and cost
| Contender | Version | Runs | Mean positions | Mean model calls | Stopped by a limit | Median / max |M² error| | Cost |
|---|---|---|---|---|---|---|---|
| agent | v1 | 26 | 14.3 | 21.4 | A8, C2 | 0.2 % / 6.1 % | $2.05 |
| agent | v2 | 26 | 15.5 | 16.2 | none | 0.2 % / 107.8 % | $2.05 |
| agent-noview | v1 | 5 | 17.2 | 23.8 | none | 3.6 % / 109.1 % | $0.51 |
| agent-noview | v2 | 5 | 15.0 | 17.4 | none | 2.6 % / 109.0 % | $0.32 |
| scripted | v1 | 26 | 15.8 | – | – | 0.3 % / 108.7 % | $0.00 |
| scripted | v2 | 26 | 16.5 | – | – | 0.2 % / 110.7 % | $0.00 |
| naive | v1 | 26 | 10.0 | – | – | 2.3 % / 104.3 % | $0.00 |
| naive | v2 | 26 | 10.0 | – | – | 2.8 % / 123.6 % | $0.00 |
Constraint violations in group B
Measurements that broke the operator note (B2: 1 if more than 12 positions were used; B3 has no constraint).
| Contender | B1 v1 · v2 | B2 v1 · v2 | B3 v1 · v2 | B4 v1 · v2 | B5 v1 · v2 | Total |
|---|---|---|---|---|---|---|
| agent | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | 0 |
| scripted | 1 · 1 | 1 · 1 | 0 · 0 | 1 · 1 | 5 · 5 | 16 |
| naive | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | 4 · 4 | 8 |
Main failure modes
C1, ghost reflection at every position (seen): saw the ghost but kept the points. In v1 the agent with view_frame named the ghost and gave the right verdict. In v2 it looked at frames only after coverage was met, described the second spot, and still kept those points. Without view_frame the agent read the constant axes_not_aligned flag as a rotated beam. The wrong runs report a confident, very wrong M².
| Version | Contender | Verdict (expected) | Correct | M² err x / y | Views | Fault named |
|---|---|---|---|---|---|---|
| v1 | agent | measurement_failed (measurement_failed) | yes | – / – | 4 | yes |
| v1 | agent-noview | iso_compliant (measurement_failed) | no | +109.1 % / +59.6 % | 0 | no |
| v2 | agent | iso_compliant (measurement_failed) | no | +107.8 % / +59.1 % | 3 | yes |
| v2 | agent-noview | iso_compliant (measurement_failed) | no | +109.0 % / +59.8 % | 0 | no |
C5, iris clipping the beam over 30-70 mm (no flag fires): never looked. In no run did the agent look at a frame. It accepted far points inside the iris range and did not act on the rising fit residual, so the verdict was right but M² missed the 3 % tolerance.
| Version | Contender | Verdict (expected) | Correct | M² err x / y | Views | Fault named |
|---|---|---|---|---|---|---|
| v1 | agent | iso_compliant (iso_compliant) | no | -5.9 % / -6.1 % | 0 | no |
| v1 | agent-noview | iso_compliant (iso_compliant) | no | -7.4 % / -7.3 % | 0 | no |
| v2 | agent | iso_compliant (iso_compliant) | no | -6.3 % / -6.3 % | 0 | no |
| v2 | agent-noview | iso_compliant (iso_compliant) | no | -6.0 % / -6.1 % | 0 | no |
Total: 62 agent runs, $4.93 estimated API cost.