| # Evaluation |
|
|
| This document describes **how** each number in [`BENCHMARKS.md`](BENCHMARKS.md) was produced, and |
| enforces the project's evaluation-honesty rules. It is deliberately conservative: a metric that was |
| not measured is tagged `NOT RUN`, and a metric that got worse is shown getting worse. |
|
|
| **Status tags:** `VERIFIED` · `MEASURED` · `NOT RUN` · `OPEN` · `REJECTED`. |
|
|
| --- |
|
|
| ## 1. Evaluation-honesty rules (enforced by convention and by tooling) |
|
|
| 1. **Evidence before claims.** Every reported number has an artifact path. A number with no artifact |
| does not appear in the docs. |
| 2. **Two protocols are never collapsed.** Grounding is reported under canonical **and** matched6. |
| 3. **Two test sets are never collapsed.** Change-VQA is reported on `test` **and** `test2`. |
| 4. **accuracy never travels without macro-F1** for imbalanced multi-class heads. |
| 5. **Validation is not test.** The router figure is labelled validation, ungated, n = 86. |
| 6. **A negative result stays negative.** Calibration ECE worsened; it is shown worsening. |
| 7. **USABLE ≠ ACCEPTED.** The VLM adapter's metrics are real; its status is acceptance-rejected. |
| 8. **No composite/vanity score.** There is no single headline accuracy for the system, and none is |
| invented by averaging the per-task numbers. |
|
|
| ## 2. Per-task evaluation protocols |
|
|
| ### 2.1 Change detection — `VERIFIED` |
|
|
| - **Split:** LEVIR-CD-256 test, n = 2,048 (immutable public split). |
| - **Threshold:** 0.50, frozen in config. |
| - **Metrics:** pooled IoU / macro IoU / pooled F1, plus the full confusion counts so any metric can |
| be recomputed. |
| - **Why both pooled and macro:** the test split is only ≈ 5 % changed pixels |
| (mean change fraction 0.0509). Pooled IoU (0.8122) and macro IoU (0.8457) answer different |
| questions about that imbalance. |
| - **Artifact:** `artifacts/change/eval_test/eval_result.json`. |
|
|
| ### 2.2 Grounding — `MEASURED`, two protocols × two decode variants |
|
|
| - **Split:** VRSBench, n = 16,159. |
| - **Box convention:** VRSBench 0–100 → project 0–1, via declared `benchmark_box_scale: 100.0`. |
| - **Protocols:** *canonical* and *matched6* — both reported. |
| - **Decode variants:** *head threshold* (the shipped decode), *head argmax*, and *zero-shot |
| matched* (baseline). |
| - **Resolution:** frozen at 224; 448 rejected by a pre-registered paired test. |
|
|
| | Protocol | decode | mean best IoU | recall@0.5 | |
| |---|---|---|---| |
| | canonical | head threshold | 0.2838 | 0.2198 | |
| | canonical | head argmax | 0.1215 | — | |
| | canonical | zero-shot matched | 0.0972 | — | |
| | matched6 | head threshold | 0.2566 | 0.1938 | |
|
|
| **Artifacts:** `…/eval_result_canonical.json`, `…/eval_result_matched6.json`. |
|
|
| ### 2.3 Optical-SAR fusion — `MEASURED`, ruling `OPEN` |
|
|
| - **Split:** held-out test, n = 4,000. |
| - **Label space:** 19 CLC classes; 14 present, 5 absent in the scored split. |
| - **macro-F1 denominator:** all 19 classes (absent classes contribute 0.0) — recorded explicitly. |
| - **Pre-registration:** the metric is a pre-registered protocol (`pre_registered_11.5`); |
| `is_deciding_statistic: False`. |
| - **Artifact:** `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`. |
|
|
| ### 2.4 Change-VQA — `MEASURED`, ruling `OPEN` |
|
|
| - **Splits:** `test` (n = 39,686) and `test2`. |
| - **Selection:** epoch 8, chosen on val answer accuracy 0.700018. |
| - **Artifact:** `artifacts/change_vqa/run/PROMOTION.json`. |
|
|
| ### 2.5 VLM adapter — `MEASURED`, `ACCEPTANCE-REJECTED` |
|
|
| - **Split:** frozen 1,000-question subset. |
| - **Metrics:** exact_match 0.963, F1 0.96432 (+49.5 pp over unadapted baseline). |
| - **Decision:** the artifact is **not accepted** for promotion; the record of *why* is stored in |
| `artifacts/vlm/phase6_closure.json` under `why_acceptance_rejected`. |
| - **Artifact:** `artifacts/vlm/phase6_closure.json` (`status: CLOSED`). |
|
|
| ### 2.6 Router — `MEASURED`, `TEST NOT RUN` |
|
|
| - **Split scored:** validation, n = 86, corpus-limited. |
| - **Metric:** overall **ungated** accuracy 0.965116. |
| - **Not done:** the test split was never scored. The val number is indicative only. |
|
|
| ### 2.7 Calibration — `MEASURED`, not an improvement |
|
|
| - **Fit split:** val, n = 16,441. Temperature **T = 0.9772732**. |
| - **Result:** ECE 0.013755 → **0.014929** (`ece_improvement = −0.001174`). |
| - **Retention rationale:** part of the frozen config, **not** because it helped. |
| - **Artifact:** `artifacts/calibration_v001.json`. |
|
|
| ## 3. Behavioural evaluation (live validation) |
|
|
| Accuracy and behaviour are evaluated separately. The deployed stack was driven in a headed browser, |
| one upload per case, with per-case screenshots and recorded run ids: |
|
|
| | Property | Result | |
| |---|---| |
| | Independent full passes | **3** | |
| | Cases per pass | 8 (6 regression + 2 router-defect) | |
| | Passes at 8/8 | **3 of 3** | |
| | Live runs | **24** | |
| | Correct dispatches | **24** | |
| | Mock-node contamination | **0** | |
| | Trace fill | **94.4444 %** | |
| | Frontend regression suite | **106 passed** | |
|
|
| ### 3.1 Harness integrity — a bug that was caught |
|
|
| An earlier harness revision typed queries with **synthetic CDP key events**, which Chrome **silently |
| drops when the window lacks OS focus**. The harness therefore dispatched the page's *default* query |
| and still recorded a "result" — a false pass. |
|
|
| The current harness **asserts form state before dispatch**: that the query box really holds the |
| intended query, that `#obsTail` reads `ready`, and that both frames are attached for pair tasks. The |
| earlier 8/8 run was independently checked and confirmed **not** to have been infected (its answers |
| were query-specific and the query text was embedded in the answers). This failure mode is recorded |
| here because it is exactly the kind of silent false-positive an evaluation harness must not have. |
|
|
| ## 4. Test suite |
|
|
| | Suite | Result | Notes | |
| |---|---|---| |
| | `tests/unit/test_frontend_live_wiring.py` | **106 passed** | re-run this session | |
| | Doc/frontend suite (5 files) | **183 passed** | | |
| | Full `tests/unit` | 5–6 failures, **all environmental/ordering** | 4× `test_safe_delete_shim` (sandbox delete guard), 1 ordering flake (passes in isolation), 1 stale adapter test (CROMA now shipped) | |
| | Affected files re-run together | **137 passed** | confirms the failures are not real regressions | |
|
|
| **The environmental failures are not hidden.** They are attributable to the sandbox delete guard and |
| test ordering, not to the code under test. |
|
|
| ## 5. What is NOT evaluated |
|
|
| | Item | State | |
| |---|---| |
| | System-level end-to-end accuracy | **NOT RUN — none exists** | |
| | Router test split | **NOT RUN** | |
| | Benchmark adapters | **NOT RUN** | |
| | End-to-end latency benchmark | **NOT RUN** | |
| | Cross-dataset generalisation | **NOT RUN** | |
| | Human evaluation | **NOT RUN** | |
| | Adversarial / robustness evaluation | **NOT RUN** | |
|
|
| ## 6. Reproducing the metric check |
|
|
| ```bash |
| python release/tools/verify_readme_metrics.py |
| ``` |
|
|
| This walks every claim in [`../README.md`](../README.md) and [`BENCHMARKS.md`](BENCHMARKS.md) to its |
| source artifact and compares values at the printed precision. It prints `ALL CLAIMS VERIFIED` (exit 0) |
| only when **all 20** numeric claims match and the status assertions hold. Committed output: |
| `release/tools/readme_metrics_report.txt`. |
|
|
