| # BeamPilot evaluation summary (v1 + v2) | |
| Generated by `scripts/make_evaluation_summary.py` from `docs/results/evaluation-v1.jsonl` and `docs/results/evaluation-v2.jsonl`. No number on this page is typed by hand. | |
| ## Plans | |
| - **v1:** [evaluation-plan.md](evaluation-plan.md), pre-registered in `d2f445e` (plan), `b3025e9` (deviations 1-7), `9ca92df` (deviation 8 and evaluation script), `cda0f73` (deviation 9); run on `main` at `03fb2c1`. Full results: [evaluation-results.md](evaluation-results.md). | |
| - **v2:** [evaluation-plan-v2.md](evaluation-plan-v2.md), pre-registered in `75fe382` (plan), `3241612` (deviations 1-2); run on `main` at `3403a76`. Full results: [evaluation-results-v2.md](evaluation-results-v2.md). | |
| Between v1 and v2 only two things changed: the model-call limit (40 → 60) and a prompt asking for parallel tool calls. Seeds differ. | |
| ## Read this first | |
| - **Small N.** Each case ran twice in total: once in v1 and once in v2, with different seeds. A difference of one or two cases is noise, not a measured effect. | |
| - **C1 is "seen".** The prompt was changed after the agent failed the same fault type during development, so C1 is never counted as held out. C2-C5 were held out in v1 and seen in v2. | |
| - **The scripted baseline cannot read operator notes, by design.** In group B it runs without the note; its violations are counted below. Group B is therefore not a fair contest. | |
| - **The naive scan is a reference only** (10 fixed positions, 1 ms). | |
| ## Correct verdicts | |
| Correct / run, per version and both together. Group C is run twice per version for the agent: with `view_frame` (`agent`) and without (`agent-noview`). | |
| | Contender | A: v1 · v2 · both | B: v1 · v2 · both | C2-C5: v1 · v2 · both | C1 (seen): v1 · v2 · both | | |
| |---|---|---|---|---| | |
| | agent | 15/16 · 16/16 · **31/32** | 4/5 · 4/5 · **8/10** | 2/4 · 3/4 · **5/8** | 1/1 · 0/1 · **1/2** | | |
| | agent-noview | – | – | 2/4 · 3/4 · **5/8** | 0/1 · 0/1 · **0/2** | | |
| | scripted | 16/16 · 14/16 · **30/32** | 5/5 · 5/5 · **10/10** | 2/4 · 3/4 · **5/8** | 0/1 · 0/1 · **0/2** | | |
| | naive | 2/16 · 1/16 · **3/32** | 0/5 · 0/5 · **0/10** | 0/4 · 0/4 · **0/8** | 0/1 · 0/1 · **0/2** | | |
| ## Effort, accuracy and cost | |
| | Contender | Version | Runs | Mean positions | Mean model calls | Stopped by a limit | Median / max \|M² error\| | Cost | | |
| |---|---|---|---|---|---|---|---| | |
| | agent | v1 | 26 | 14.3 | 21.4 | A8, C2 | 0.2 % / 6.1 % | $2.05 | | |
| | agent | v2 | 26 | 15.5 | 16.2 | none | 0.2 % / 107.8 % | $2.05 | | |
| | agent-noview | v1 | 5 | 17.2 | 23.8 | none | 3.6 % / 109.1 % | $0.51 | | |
| | agent-noview | v2 | 5 | 15.0 | 17.4 | none | 2.6 % / 109.0 % | $0.32 | | |
| | scripted | v1 | 26 | 15.8 | – | – | 0.3 % / 108.7 % | $0.00 | | |
| | scripted | v2 | 26 | 16.5 | – | – | 0.2 % / 110.7 % | $0.00 | | |
| | naive | v1 | 26 | 10.0 | – | – | 2.3 % / 104.3 % | $0.00 | | |
| | naive | v2 | 26 | 10.0 | – | – | 2.8 % / 123.6 % | $0.00 | | |
| ## Constraint violations in group B | |
| Measurements that broke the operator note (B2: 1 if more than 12 positions were used; B3 has no constraint). | |
| | Contender | B1 v1 · v2 | B2 v1 · v2 | B3 v1 · v2 | B4 v1 · v2 | B5 v1 · v2 | Total | | |
| |---|---|---|---|---|---|---| | |
| | agent | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | **0** | | |
| | scripted | 1 · 1 | 1 · 1 | 0 · 0 | 1 · 1 | 5 · 5 | **16** | | |
| | naive | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | 4 · 4 | **8** | | |
| ## Main failure modes | |
| **C1, ghost reflection at every position (seen): saw the ghost but kept the points.** In v1 the agent with `view_frame` named the ghost and gave the right verdict. In v2 it looked at frames only after coverage was met, described the second spot, and still kept those points. Without `view_frame` the agent read the constant `axes_not_aligned` flag as a rotated beam. The wrong runs report a confident, very wrong M². | |
| | Version | Contender | Verdict (expected) | Correct | M² err x / y | Views | Fault named | | |
| |---|---|---|---|---|---|---| | |
| | v1 | agent | measurement_failed (measurement_failed) | yes | – / – | 4 | yes | | |
| | v1 | agent-noview | iso_compliant (measurement_failed) | **no** | +109.1 % / +59.6 % | 0 | no | | |
| | v2 | agent | iso_compliant (measurement_failed) | **no** | +107.8 % / +59.1 % | 3 | yes | | |
| | v2 | agent-noview | iso_compliant (measurement_failed) | **no** | +109.0 % / +59.8 % | 0 | no | | |
| **C5, iris clipping the beam over 30-70 mm (no flag fires): never looked.** In no run did the agent look at a frame. It accepted far points inside the iris range and did not act on the rising fit residual, so the verdict was right but M² missed the 3 % tolerance. | |
| | Version | Contender | Verdict (expected) | Correct | M² err x / y | Views | Fault named | | |
| |---|---|---|---|---|---|---| | |
| | v1 | agent | iso_compliant (iso_compliant) | **no** | -5.9 % / -6.1 % | 0 | no | | |
| | v1 | agent-noview | iso_compliant (iso_compliant) | **no** | -7.4 % / -7.3 % | 0 | no | | |
| | v2 | agent | iso_compliant (iso_compliant) | **no** | -6.3 % / -6.3 % | 0 | no | | |
| | v2 | agent-noview | iso_compliant (iso_compliant) | **no** | -6.0 % / -6.1 % | 0 | no | | |
| Total: 62 agent runs, $4.93 estimated API cost. | |