File size: 5,192 Bytes
0ffbb74 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | # BeamPilot evaluation summary (v1 + v2)
Generated by `scripts/make_evaluation_summary.py` from `docs/results/evaluation-v1.jsonl` and `docs/results/evaluation-v2.jsonl`. No number on this page is typed by hand.
## Plans
- **v1:** [evaluation-plan.md](evaluation-plan.md), pre-registered in `d2f445e` (plan), `b3025e9` (deviations 1-7), `9ca92df` (deviation 8 and evaluation script), `cda0f73` (deviation 9); run on `main` at `03fb2c1`. Full results: [evaluation-results.md](evaluation-results.md).
- **v2:** [evaluation-plan-v2.md](evaluation-plan-v2.md), pre-registered in `75fe382` (plan), `3241612` (deviations 1-2); run on `main` at `3403a76`. Full results: [evaluation-results-v2.md](evaluation-results-v2.md).
Between v1 and v2 only two things changed: the model-call limit (40 → 60) and a prompt asking for parallel tool calls. Seeds differ.
## Read this first
- **Small N.** Each case ran twice in total: once in v1 and once in v2, with different seeds. A difference of one or two cases is noise, not a measured effect.
- **C1 is "seen".** The prompt was changed after the agent failed the same fault type during development, so C1 is never counted as held out. C2-C5 were held out in v1 and seen in v2.
- **The scripted baseline cannot read operator notes, by design.** In group B it runs without the note; its violations are counted below. Group B is therefore not a fair contest.
- **The naive scan is a reference only** (10 fixed positions, 1 ms).
## Correct verdicts
Correct / run, per version and both together. Group C is run twice per version for the agent: with `view_frame` (`agent`) and without (`agent-noview`).
| Contender | A: v1 · v2 · both | B: v1 · v2 · both | C2-C5: v1 · v2 · both | C1 (seen): v1 · v2 · both |
|---|---|---|---|---|
| agent | 15/16 · 16/16 · **31/32** | 4/5 · 4/5 · **8/10** | 2/4 · 3/4 · **5/8** | 1/1 · 0/1 · **1/2** |
| agent-noview | – | – | 2/4 · 3/4 · **5/8** | 0/1 · 0/1 · **0/2** |
| scripted | 16/16 · 14/16 · **30/32** | 5/5 · 5/5 · **10/10** | 2/4 · 3/4 · **5/8** | 0/1 · 0/1 · **0/2** |
| naive | 2/16 · 1/16 · **3/32** | 0/5 · 0/5 · **0/10** | 0/4 · 0/4 · **0/8** | 0/1 · 0/1 · **0/2** |
## Effort, accuracy and cost
| Contender | Version | Runs | Mean positions | Mean model calls | Stopped by a limit | Median / max \|M² error\| | Cost |
|---|---|---|---|---|---|---|---|
| agent | v1 | 26 | 14.3 | 21.4 | A8, C2 | 0.2 % / 6.1 % | $2.05 |
| agent | v2 | 26 | 15.5 | 16.2 | none | 0.2 % / 107.8 % | $2.05 |
| agent-noview | v1 | 5 | 17.2 | 23.8 | none | 3.6 % / 109.1 % | $0.51 |
| agent-noview | v2 | 5 | 15.0 | 17.4 | none | 2.6 % / 109.0 % | $0.32 |
| scripted | v1 | 26 | 15.8 | – | – | 0.3 % / 108.7 % | $0.00 |
| scripted | v2 | 26 | 16.5 | – | – | 0.2 % / 110.7 % | $0.00 |
| naive | v1 | 26 | 10.0 | – | – | 2.3 % / 104.3 % | $0.00 |
| naive | v2 | 26 | 10.0 | – | – | 2.8 % / 123.6 % | $0.00 |
## Constraint violations in group B
Measurements that broke the operator note (B2: 1 if more than 12 positions were used; B3 has no constraint).
| Contender | B1 v1 · v2 | B2 v1 · v2 | B3 v1 · v2 | B4 v1 · v2 | B5 v1 · v2 | Total |
|---|---|---|---|---|---|---|
| agent | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | **0** |
| scripted | 1 · 1 | 1 · 1 | 0 · 0 | 1 · 1 | 5 · 5 | **16** |
| naive | 0 · 0 | 0 · 0 | 0 · 0 | 0 · 0 | 4 · 4 | **8** |
## Main failure modes
**C1, ghost reflection at every position (seen): saw the ghost but kept the points.** In v1 the agent with `view_frame` named the ghost and gave the right verdict. In v2 it looked at frames only after coverage was met, described the second spot, and still kept those points. Without `view_frame` the agent read the constant `axes_not_aligned` flag as a rotated beam. The wrong runs report a confident, very wrong M².
| Version | Contender | Verdict (expected) | Correct | M² err x / y | Views | Fault named |
|---|---|---|---|---|---|---|
| v1 | agent | measurement_failed (measurement_failed) | yes | – / – | 4 | yes |
| v1 | agent-noview | iso_compliant (measurement_failed) | **no** | +109.1 % / +59.6 % | 0 | no |
| v2 | agent | iso_compliant (measurement_failed) | **no** | +107.8 % / +59.1 % | 3 | yes |
| v2 | agent-noview | iso_compliant (measurement_failed) | **no** | +109.0 % / +59.8 % | 0 | no |
**C5, iris clipping the beam over 30-70 mm (no flag fires): never looked.** In no run did the agent look at a frame. It accepted far points inside the iris range and did not act on the rising fit residual, so the verdict was right but M² missed the 3 % tolerance.
| Version | Contender | Verdict (expected) | Correct | M² err x / y | Views | Fault named |
|---|---|---|---|---|---|---|
| v1 | agent | iso_compliant (iso_compliant) | **no** | -5.9 % / -6.1 % | 0 | no |
| v1 | agent-noview | iso_compliant (iso_compliant) | **no** | -7.4 % / -7.3 % | 0 | no |
| v2 | agent | iso_compliant (iso_compliant) | **no** | -6.3 % / -6.3 % | 0 | no |
| v2 | agent-noview | iso_compliant (iso_compliant) | **no** | -6.0 % / -6.1 % | 0 | no |
Total: 62 agent runs, $4.93 estimated API cost.
|