thundercode
/

SatQuery

+# Benchmarks
+**Every number here is artifact-backed.** Where a metric exists, its source file is named. Where a
+benchmark does **not** exist, that is stated explicitly and tagged `NOT RUN` — an absent number is
+never silently omitted or replaced with an estimate.
+**Status tags:** `VERIFIED` · `MEASURED` · `NOT RUN` · `OPEN` · `REJECTED`.
+---
+## 1. Headline table
+| Capability | Metric | Value | Split / protocol | Status |
+|---|---|---|---|---|
+| Change detection | pooled IoU | **0.8122** | LEVIR-CD-256 test, n = 2048, thr 0.50 | **VERIFIED** |
+| Change detection | macro IoU | **0.8457** | same | **VERIFIED** |
+| Change detection | pooled F1 | **0.8964** | same | **VERIFIED** |
+| Grounding | mean best IoU | **0.2838** / **0.2566** | VRSBench n = 16159, canonical / matched6 | MEASURED (2 protocols) |
+| Grounding | recall@0.5 | **0.2198** / **0.1938** | same | MEASURED (2 protocols) |
+| Grounding | head-argmax IoU | **0.1215** | canonical | MEASURED |
+| Grounding | zero-shot baseline IoU | **0.0972** | canonical | MEASURED (baseline) |
+| Optical-SAR fusion | accuracy | **0.931** | held-out test n = 4000, 19 classes | MEASURED, ruling **OPEN** |
+| Optical-SAR fusion | macro F1 | **0.434161** | same | MEASURED, ruling **OPEN** |
+| Change-VQA | accuracy | **0.697626** | test n = 39686 | MEASURED, ruling **OPEN** |
+| Change-VQA | macro F1 | **0.378373** | same | MEASURED, ruling **OPEN** |
+| Change-VQA | accuracy (2nd set) | **0.651469** | test2 | MEASURED, ruling **OPEN** |
+| Change-VQA | macro F1 (2nd set) | **0.372309** | test2 | MEASURED, ruling **OPEN** |
+| VLM (adapted) | exact_match | **0.963** | frozen 1000-question subset | MEASURED, **ACCEPTANCE-REJECTED** |
+| VLM (adapted) | F1 | **0.96432** | same | MEASURED, **ACCEPTANCE-REJECTED** |
+| Router | overall **ungated** accuracy | **0.965116** | val n = 86, corpus-limited | MEASURED — **TEST NOT RUN** |
+| Calibration | ECE before / after | **0.013755 → 0.014929** | val n = 16441, T = 0.9773 | MEASURED — **worse** |
+### Source artifacts
+| Metric family | Artifact |
+|---|---|
+| change | `artifacts/change/eval_test/eval_result.json` |
+| grounding | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json`, `…_matched6.json` |
+| optical-SAR | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` |
+| change-VQA | `artifacts/change_vqa/run/PROMOTION.json` |
+| router | `artifacts/router/threshold_sweep_val.json` |
+| calibration | `artifacts/calibration_v001.json` |
+| VLM | `artifacts/vlm/phase6_closure.json` |
+> All 20 values in the table above are checked against these files by
+> [`../tools/verify_readme_metrics.py`](../tools/verify_readme_metrics.py). Its output
+> (`ALL CLAIMS VERIFIED`) is committed as `../tools/readme_metrics_report.txt`.
+## 2. Rules this table follows
+1. **Two protocols are never collapsed.** Grounding is reported under *both* the canonical and
+   matched6 protocols. Quoting 0.2838 alone would be selective.
+2. **Two test sets are never collapsed.** Change-VQA is reported on `test` **and** `test2`.
+3. **accuracy never travels without macro-F1.** For imbalanced multi-class heads (optical-SAR,
+   change-VQA) the macro-F1 is reported alongside accuracy, always.
+4. **Validation is not test.** The router number is labelled "overall **ungated** accuracy", val,
+   n = 86. It is not a test result.
+5. **A negative result stays negative.** Calibration ECE worsened and is shown worsening.
+6. **USABLE ≠ ACCEPTED.** The VLM metrics are real; the artifact is nevertheless acceptance-rejected.
+## 3. What is NOT benchmarked
+| Benchmark | Status | Note |
+|---|---|---|
+| **System-level end-to-end accuracy** | **NOT RUN — none exists** | There is no measured end-to-end benchmark of the full router→specialist→envelope pipeline. No such number is claimed anywhere. |
+| **Router test split** | **NOT RUN** | Only the validation split (n = 86) was scored. |
+| **Benchmark adapters** | **NOT RUN** | Adapter-based benchmark runs were not executed. |
+| **Efficiency / latency benchmark** | not systematically measured | Per-specialist latency is recorded incidentally in artifacts (e.g. grounding `latency_ms_per_image` 2.205 ms for the head), but there is no end-to-end latency benchmark. |
+| **Cross-dataset generalisation** | **NOT RUN** | Each specialist is evaluated only on its own training-family test split. |
+## 4. Live validation (behavioural, not accuracy)
+Accuracy is separate from **behavioural** validation. The deployed stack was driven end-to-end in a
+headed browser, one upload per case, with per-case screenshots and recorded run ids:
+| Property | Result |
+|---|---|
+| Independent full passes | **3** |
+| Cases per pass | 8 (6 regression + 2 router-defect) |
+| Passes at 8/8 | **3 of 3** |
+| Live runs executed | **24** |
+| Correct dispatches | **24** |
+| Mock-node contamination | **0** on every live run |
+| Trace fill | **94.4444 %** on every live run |
+| Frontend regression suite | **106 passed** (`tests/unit/test_frontend_live_wiring.py`) |
+Every pass produced **fresh run identifiers** — no run id is shared between passes. This is
+behavioural evidence that the pipeline runs and routes correctly; it is **not** an accuracy claim.
+## 5. How to reproduce
+```bash
+# verify every README/benchmark number against its artifact
+python release/tools/verify_readme_metrics.py
+```
+The script resolves nested artifact keys (including keys that themselves contain dots, such as the
+`recall` dict keyed `"0.10"/"0.25"/"0.50"`) and compares each value at the precision printed in the
+README. It exits non-zero if any claim fails. See [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) for the
+full reproduction guide.