release: add docs/BENCHMARKS.md

docs/BENCHMARKS.md

@@ -1,77 +1,535 @@
-# Benchmarks
+# Benchmarks — deep reference
 
-**Every number here is artifact-backed.** Where a metric exists, its source file is named. Where a
-benchmark does **not** exist, that is stated explicitly and tagged `NOT RUN` — an absent number is
-never silently omitted or replaced with an estimate.
+**Every number here is artifact-backed.** Where a metric exists, its source file and the **exact key
+path** inside that file are named. Where a benchmark does **not** exist, that is stated explicitly and
+tagged `NOT RUN` — an absent number is never silently omitted, never estimated, and never replaced with
+a plausible-looking substitute.
 
-**Status tags:** `VERIFIED` · `MEASURED` · `NOT RUN` · `OPEN` · `REJECTED`.
+**Status tags:** `VERIFIED` · `MEASURED` · `NOT RUN` · `OPEN` · `REJECTED` · `BLOCKED`.
+
+**The single most important rule in this document: do not fabricate.** If a value is not in an
+artifact that was read, it does not appear here. Where a fact is not established, this document writes
+`UNKNOWN — not established from the available evidence`.
+
+> **There is no system-level end-to-end benchmark.** The router → specialist → envelope pipeline has
+> never been scored end-to-end, and no end-to-end accuracy is claimed anywhere in this repository. What
+> exists is (a) per-specialist metrics on their own training-family test splits, and (b) a behavioural
+> live-validation record (§5) that proves the pipeline *runs and routes*, not that it is *accurate*.
+
+---
+
+## Table of contents
+
+1. [Headline table](#1-headline-table)
+2. [Source artifacts](#2-source-artifacts)
+3. [The honesty rules this document follows](#3-the-honesty-rules-this-document-follows)
+4. [Per-task evaluation protocols](#4-per-task-evaluation-protocols)
+   - 4.1 [Change detection](#41-change-detection--verified)
+   - 4.2 [Grounding](#42-grounding--measured-two-protocols--two-decode-variants)
+   - 4.3 [Optical-SAR fusion](#43-optical-sar-fusion--measured-ruling-open)
+   - 4.4 [Change-VQA](#44-change-vqa--measured-ruling-open)
+   - 4.5 [VLM adapter](#45-vlm-adapter--measured-acceptance-rejected)
+   - 4.6 [Router](#46-router--measured-test-not-run)
+   - 4.7 [Calibration](#47-calibration--measured-negative-result)
+5. [Live validation — behavioural, not accuracy](#5-live-validation--behavioural-not-accuracy)
+6. [What is NOT benchmarked](#6-what-is-not-benchmarked)
+7. [How to reproduce](#7-how-to-reproduce)
+8. [The verification tool — exactly what it checks](#8-the-verification-tool--exactly-what-it-checks)
+9. [Artifact schemas and key-path resolution](#9-artifact-schemas-and-key-path-resolution)
+10. [Metric glossary and aggregation arithmetic](#10-metric-glossary-and-aggregation-arithmetic)
+11. [Evidence index](#11-evidence-index)
 
 ---
 
 ## 1. Headline table
 
-| Capability | Metric | Value | Split / protocol | Status |
-|---|---|---|---|---|
-| Change detection | pooled IoU | **0.8122** | LEVIR-CD-256 test, n = 2048, thr 0.50 | **VERIFIED** |
-| Change detection | macro IoU | **0.8457** | same | **VERIFIED** |
-| Change detection | pooled F1 | **0.8964** | same | **VERIFIED** |
-| Grounding | mean best IoU | **0.2838** / **0.2566** | VRSBench n = 16159, canonical / matched6 | MEASURED (2 protocols) |
-| Grounding | recall@0.5 | **0.2198** / **0.1938** | same | MEASURED (2 protocols) |
-| Grounding | head-argmax IoU | **0.1215** | canonical | MEASURED |
-| Grounding | zero-shot baseline IoU | **0.0972** | canonical | MEASURED (baseline) |
-| Optical-SAR fusion | accuracy | **0.931** | held-out test n = 4000, 19 classes | MEASURED, ruling **OPEN** |
-| Optical-SAR fusion | macro F1 | **0.434161** | same | MEASURED, ruling **OPEN** |
-| Change-VQA | accuracy | **0.697626** | test n = 39686 | MEASURED, ruling **OPEN** |
-| Change-VQA | macro F1 | **0.378373** | same | MEASURED, ruling **OPEN** |
-| Change-VQA | accuracy (2nd set) | **0.651469** | test2 | MEASURED, ruling **OPEN** |
-| Change-VQA | macro F1 (2nd set) | **0.372309** | test2 | MEASURED, ruling **OPEN** |
-| VLM (adapted) | exact_match | **0.963** | frozen 1000-question subset | MEASURED, **ACCEPTANCE-REJECTED** |
-| VLM (adapted) | F1 | **0.96432** | same | MEASURED, **ACCEPTANCE-REJECTED** |
-| Router | overall **ungated** accuracy | **0.965116** | val n = 86, corpus-limited | MEASURED — **TEST NOT RUN** |
-| Calibration | ECE before / after | **0.013755 → 0.014929** | val n = 16441, T = 0.9773 | MEASURED — **worse** |
-
-### Source artifacts
-
-| Metric family | Artifact |
+Every row names its source artifact and the exact key path within it. The `n` and `split` columns are
+part of the claim, not decoration: a metric without its population is not a result.
+
+| Capability | Metric | Value | Split / protocol | n | Source artifact → key path | Status |
+|---|---|---|---|---|---|---|
+| Change detection | pooled IoU | **0.8122** | LEVIR-CD-256 test, thr 0.50 | 2,048 | `artifacts/change/eval_test/eval_result.json` → `metrics.pooled.iou` | **VERIFIED** |
+| Change detection | macro IoU | **0.8457** | same | 2,048 | `��/eval_result.json` → `metrics.macro.miou` | **VERIFIED** |
+| Change detection | pooled F1 | **0.8964** | same | 2,048 | `…/eval_result.json` → `metrics.pooled.f1` | **VERIFIED** |
+| Grounding | mean best IoU | **0.2838** | VRSBench, canonical (head threshold, top_k 20) | 16,159 | `…/eval_result_canonical.json` → `results.head_threshold.mean_best_iou` | MEASURED (2 protocols) |
+| Grounding | recall@0.5 | **0.2198** | canonical | 16,159 | `…/eval_result_canonical.json` → `results.head_threshold.recall.0.50` | MEASURED (2 protocols) |
+| Grounding | mean best IoU | **0.2566** | VRSBench, matched6 (head threshold, top_k 6) | 16,159 | `…/eval_result_matched6.json` → `results.head_threshold.mean_best_iou` | MEASURED (2 protocols) |
+| Grounding | recall@0.5 | **0.1938** | matched6 | 16,159 | `…/eval_result_matched6.json` → `results.head_threshold.recall.0.50` | MEASURED (2 protocols) |
+| Grounding | head-argmax IoU | **0.1215** | canonical (argmax decode) | 16,159 | `…/eval_result_canonical.json` → `results.head_argmax.mean_best_iou` | MEASURED |
+| Grounding | zero-shot baseline IoU | **0.0972** | canonical (no trained head) | 16,159 | `…/eval_result_canonical.json` → `results.zero_shot_matched.mean_best_iou` | MEASURED (baseline) |
+| Optical-SAR fusion | accuracy | **0.931** | held-out test, 19 classes | 4,000 | `…/fusion_head_production_v001/pre_registered_115_metric.json` → `accuracy` | MEASURED, ruling **OPEN** |
+| Optical-SAR fusion | macro F1 | **0.434161** | same | 4,000 | `…/pre_registered_115_metric.json` → `macro_f1` | MEASURED, ruling **OPEN** |
+| Change-VQA | accuracy | **0.697626** | `test` | 39,686 | `artifacts/change_vqa/run/PROMOTION.json` → `verification.test_accuracy` | MEASURED, ruling **OPEN** |
+| Change-VQA | macro F1 | **0.378373** | `test` | 39,686 | `…/PROMOTION.json` → `verification.test_macro_f1` | MEASURED, ruling **OPEN** |
+| Change-VQA | accuracy (2nd set) | **0.651469** | `test2` | 31,036 | `…/PROMOTION.json` → `verification.test2_accuracy` | MEASURED, ruling **OPEN** |
+| Change-VQA | macro F1 (2nd set) | **0.372309** | `test2` | 31,036 | `…/PROMOTION.json` → `verification.test2_macro_f1` | MEASURED, ruling **OPEN** |
+| VLM (adapted) | exact_match | **0.963** | frozen 1,000-question subset | 1,000 | `artifacts/vlm/phase6_closure.json` → `why_usable_verified.adapted_test.exact_match` | MEASURED, **ACCEPTANCE-REJECTED** |
+| VLM (adapted) | F1 | **0.96432** | same | 1,000 | `…/phase6_closure.json` → `why_usable_verified.adapted_test.f1` | MEASURED, **ACCEPTANCE-REJECTED** |
+| Router | overall **ungated** accuracy | **0.965116** | val, corpus-limited | 86 | `artifacts/router/threshold_sweep_val.json` → `overall_ungated_accuracy` | MEASURED — **TEST NOT RUN** |
+| Calibration | ECE before / after | **0.013755 → 0.014929** | val, T = 0.9773 | 16,441 | `artifacts/calibration_v001.json` → `metrics.ece_before` / `metrics.ece_after` | MEASURED — **worse** |
+
+### 1.1 The three facts most likely to be misread
+
+1. **Grounding is reported under two protocols and two decode variants.** The trained head scores
+   **0.2838** (canonical) and **0.2566** (matched6) mean best IoU; the head-argmax decode scores
+   **0.1215** and the zero-shot baseline **0.0972**. Quoting one cell of that grid alone is selective.
+   The **matched6** protocol is the decode-matched comparison (head emits 6 boxes, baseline emits
+   5.99); the **canonical** protocol is the config default (top_k 20).
+2. **Change-VQA is reported on two test sets.** `test` (0.697626 / 0.378373) and `test2`
+   (0.651469 / 0.372309). Quoting only the better one would be selective.
+3. **Router 0.965116 is validation, ungated, n = 86.** It is not a test result; the router test split
+   was **NOT RUN** for the shipped artifact.
+
+### 1.2 The only `VERIFIED` headline
+
+The change-detection triple — **pooled IoU 0.8122 / macro IoU 0.8457 / pooled F1 0.8964** — is the
+**only** headline in this repository that carries the `VERIFIED` tag, because it is the only one
+measured against a single, immutable public test split with a frozen threshold. Everything else is
+`MEASURED` (real, artifact-backed, but with an open ruling, a protocol caveat, or a validation-only
+population).
+
+---
+
+## 2. Source artifacts
+
+| Metric family | Artifact (paths relative to the source repository root) |
 |---|---|
 | change | `artifacts/change/eval_test/eval_result.json` |
-| grounding | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json`, `…_matched6.json` |
+| grounding (canonical) | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json` |
+| grounding (matched6) | `artifacts/grounding/remoteclip_grounding_v001/eval_result_matched6.json` |
 | optical-SAR | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` |
 | change-VQA | `artifacts/change_vqa/run/PROMOTION.json` |
 | router | `artifacts/router/threshold_sweep_val.json` |
 | calibration | `artifacts/calibration_v001.json` |
 | VLM | `artifacts/vlm/phase6_closure.json` |
 
-> All 20 values in the table above are checked against these files by
+Each of these artifacts carries the frozen config hash `78f1e3700da15aa1` (where it records a hash at
+all), which ties it to the same `configs/base.yaml`. A config edit moves the hash and detaches the
+number from its configuration — which is why the change head is wired into serving via a registry
+override rather than a config key (`docs/MODELS.md` §3.1).
+
+> **All 20 numeric claims in the headline table are checked against these files by
 > [`../tools/verify_readme_metrics.py`](../tools/verify_readme_metrics.py). Its output
-> (`ALL CLAIMS VERIFIED`) is committed as `../tools/readme_metrics_report.txt`.
+> (`ALL CLAIMS VERIFIED`) is committed as [`../tools/readme_metrics_report.txt`](../tools/readme_metrics_report.txt).**
+> §8 documents exactly what that tool checks.
 
-## 2. Rules this table follows
+---
 
-1. **Two protocols are never collapsed.** Grounding is reported under *both* the canonical and
-   matched6 protocols. Quoting 0.2838 alone would be selective.
-2. **Two test sets are never collapsed.** Change-VQA is reported on `test` **and** `test2`.
-3. **accuracy never travels without macro-F1.** For imbalanced multi-class heads (optical-SAR,
-   change-VQA) the macro-F1 is reported alongside accuracy, always.
-4. **Validation is not test.** The router number is labelled "overall **ungated** accuracy", val,
-   n = 86. It is not a test result.
-5. **A negative result stays negative.** Calibration ECE worsened and is shown worsening.
-6. **USABLE ≠ ACCEPTED.** The VLM metrics are real; the artifact is nevertheless acceptance-rejected.
+## 3. The honesty rules this document follows
 
-## 3. What is NOT benchmarked
+These are not aspirations. Each one is a rule that changed what is reported, and each is checkable
+against the artifacts.
 
-| Benchmark | Status | Note |
+1. **Evidence before claims.** Every reported number has an artifact path. A number with no artifact
+   does not appear in this document.
+2. **Two protocols are never collapsed.** Grounding is reported under *both* the canonical and
+   matched6 protocols. Quoting 0.2838 alone would be selective; quoting 0.2566 alone would be too.
+3. **Two decode variants are never collapsed into the trained-head claim.** `head_argmax` (0.1215) is
+   **not** apples-to-apples with the multi-box baseline (mean best IoU is a max over predictions, so
+   1 box vs ~6 boxes flatters the head). It is reported because it is the number comparable to the
+   Phase 7 resolution experiment's zero-shot argmax — not because it decides anything.
+4. **Two test sets are never collapsed.** Change-VQA is reported on `test` **and** `test2`.
+5. **accuracy never travels without macro-F1.** For imbalanced multi-class heads (optical-SAR,
+   change-VQA) the macro-F1 is reported alongside accuracy, always, in the same table, with the same
+   split and the same `n`.
+6. **Validation is not test.** The router figure is labelled "overall **ungated** accuracy", val,
+   n = 86, corpus-limited. It is not a test result.
+7. **A negative result stays negative.** Calibration ECE worsened (0.013755 → 0.014929) and is shown
+   worsening. It is retained because it is in the frozen config, **not** because it helped.
+8. **USABLE ≠ ACCEPTED.** The VLM adapter's metrics are real and reproducible; the artifact is
+   nevertheless `ACCEPTANCE-REJECTED`. Both statements appear together.
+9. **A metric is never promoted to a system-level claim.** There is no end-to-end benchmark, and none
+   is invented by averaging the per-task numbers.
+10. **An absent number is stated, not omitted.** §6 lists everything that is `NOT RUN`.
+11. **A protocol-sensitive number carries its protocol.** Grounding always names canonical vs matched6
+    and the decode variant; optical-SAR always names the single-label subset and the 19-slot
+    denominator.
+12. **A pre-registered rule is applied as written.** The grounding resolution decision was decided by a
+    rule fixed *before* the result was seen, and the artifact records
+    `rule_changed_since_preregistration: false` (`docs/MODELS.md` §4.1).
+13. **A metric with a definitional detail states it.** The optical-SAR macro-F1 of 0.434161 is the mean
+    over **all 19 class slots**, not the 14 present classes — averaging over the present classes gives
+    0.589218, and the artifact records `macro_f1_denominator`, `classes_present`, `classes_absent` and
+    the full 19-term vector so the difference cannot be misread.
+
+---
+
+## 4. Per-task evaluation protocols
+
+### 4.1 Change detection — `VERIFIED`
+
+- **Artifact:** `artifacts/change/eval_test/eval_result.json`.
+- **Split:** LEVIR-CD-256 **test**, n = **2,048** (immutable public split).
+- **Threshold:** **0.50**, frozen in `configs/base.yaml` (`change.threshold`).
+- **Metrics:** pooled IoU / macro IoU / pooled F1, **plus the full confusion counts** so any metric can
+  be recomputed rather than trusted.
+- **Protocol detail:** tiles of 256 px (`change.tile_size: 256`), overlap 0
+  (`change.tile_overlap: 0`), minimum component 32 px (`change.min_component_pixels: 32`).
+
+**Why both pooled and macro.** The test split is only ≈ 5 % changed pixels
+(`metrics.mean_change_fraction: 0.0509`), and the **median tile has zero changed pixels**
+(`change_fraction_quantiles.p50: 0.0`). Pooled IoU (0.8122) and macro IoU (0.8457) answer different
+questions about that imbalance.
+
+**Measured numbers** (all key paths under `metrics`):
+
+| Metric | Value | Key path |
 |---|---|---|
-| **System-level end-to-end accuracy** | **NOT RUN — none exists** | There is no measured end-to-end benchmark of the full router→specialist→envelope pipeline. No such number is claimed anywhere. |
-| **Router test split** | **NOT RUN** | Only the validation split (n = 86) was scored. |
-| **Benchmark adapters** | **NOT RUN** | Adapter-based benchmark runs were not executed. |
-| **Efficiency / latency benchmark** | not systematically measured | Per-specialist latency is recorded incidentally in artifacts (e.g. grounding `latency_ms_per_image` 2.205 ms for the head), but there is no end-to-end latency benchmark. |
-| **Cross-dataset generalisation** | **NOT RUN** | Each specialist is evaluated only on its own training-family test split. |
+| pooled IoU | **0.8122** | `pooled.iou` |
+| pooled F1 | **0.8964** | `pooled.f1` |
+| pooled mean IoU (`miou`) | 0.9007 | `pooled.miou` |
+| pooled precision | 0.9195 | `pooled.precision` |
+| pooled recall | 0.8745 | `pooled.recall` |
+| macro IoU | **0.8457** | `macro.miou` |
+| macro F1 | 0.7962 | `macro.f1` |
+| macro IoU (`iou`, change-class IoU) | 0.718 | `macro.iou` |
+| macro precision | 0.8506 | `macro.precision` |
+| macro recall | 0.7757 | `macro.recall` |
+| tp | 5,978,997 | `pooled.tp` |
+| fp | 523,658 | `pooled.fp` |
+| fn | 858,407 | `pooled.fn` |
+| tn | 126,856,666 | `pooled.tn` |
+| n_pixels | 134,217,728 | `pooled.n_pixels` |
+
+**Population:** `n: 2048`, `n_images_with_change: 935`,
+`change_fraction_quantiles: {min: 0.0, p50: 0.0, p90: 0.197205, max: 0.684937}`.
+
+**Environment and integrity:** device `cuda`, torch `2.10.0+cu128`, Python `3.12.13`,
+`seconds: 55.359`; `config_hash: 78f1e3700da15aa1`, `checkpoint_config_hash: 78f1e3700da15aa1`,
+`checkpoint_config_hash_checked: true`, `config_drift: false`.
+
+**Status:** `VERIFIED`. See [`EVALUATION.md`](EVALUATION.md) §2.1 and [`MODELS.md`](MODELS.md) §3.1.
+
+---
+
+### 4.2 Grounding — `MEASURED`, two protocols × two decode variants
+
+- **Artifacts:** `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json`
+  (`head_decode.top_k: 20`) and `…/eval_result_matched6.json` (`head_decode.top_k: 6`).
+- **Split:** VRSBench **eval** split, n = **16,159** records, scored once, no sampling
+  (`n_eval_records: 16159`, `limited_run: false`).
+- **Resolution:** **224** (`resolution: 224`, `grid: 7`), frozen; 448 rejected by a pre-registered
+  paired test (`docs/MODELS.md` §4.1).
+- **Box convention:** VRSBench stores boxes normalised to **0–100**; this project stores **0–1**, via
+  the declared `grounding.benchmark_box_scale: 100.0` so the conversion cannot be applied twice.
+- **Protocols:** *canonical* and *matched6* — both reported.
+- **Decode variants:** *head threshold* (the shipped decode), *head argmax*, and *zero-shot matched*
+  (the baseline).
+
+**The `head_decode` block is recorded in each artifact** — `config_default_top_k: 20`,
+`nms_iou: 0.5`, `score_threshold: 0.4`, `top_k: 20` (canonical) / `top_k: 6` (matched6). Without it
+the artifact could not say which setting produced its number, and a re-run at the config default would
+silently yield a different figure.
+
+**The full grid — canonical** (`eval_result_canonical.json` → `results`):
+
+| Strategy | `mean_best_iou` | `recall.0.10` | `recall.0.25` | `recall.0.50` | `latency_ms_per_image` | `seconds` |
+|---|---|---|---|---|---|---|
+| `head_argmax` | **0.1215** | 0.3183 | 0.2088 | 0.0795 | 0.655 | 10.6 |
+| `head_threshold` (top_k 20) | **0.2838** | 0.6882 | 0.5047 | **0.2198** | 2.205 | 35.6 |
+| `zero_shot_matched` | **0.0972** | 0.3298 | 0.1188 | 0.0234 | — | 17.9 |
+
+**The full grid — matched6** (`eval_result_matched6.json` → `results`):
+
+| Strategy | `mean_best_iou` | `recall.0.10` | `recall.0.25` | `recall.0.50` | `latency_ms_per_image` | `seconds` |
+|---|---|---|---|---|---|---|
+| `head_argmax` | **0.1215** | 0.3183 | 0.2088 | 0.0795 | 0.652 | 10.5 |
+| `head_threshold` (top_k 6) | **0.2566** | 0.6315 | 0.4545 | **0.1938** | 2.158 | 34.9 |
+| `zero_shot_matched` | **0.0972** | 0.3298 | 0.1188 | 0.0234 | — | 15.5 |
+
+Both artifacts carry `phase7_reference: {mean_best_iou: 0.0972, recall_at_0.50: 0.0234,
+source: "docs/PHASE7_RESOLUTION_DECISION.md"}` — the recorded baseline against which the head is
+judged.
+
+**Decode-matched delta (head threshold top_k = 6 vs zero-shot): +0.1594 IoU, +0.1704 Recall@0.50.**
+The pre-registered bar was `MIN_IMPROVEMENT_IOU = 0.02`; the measured margin is **8× the bar**.
+
+**Why `head_argmax` is not the trained-head claim.** 1 box against ~6 boxes flatters the head, because
+mean best IoU takes the max over predictions. It is reported for comparability with the Phase 7
+resolution experiment, not as the head's result.
+
+**A decode defect this protocol had to fix first** (`docs/PHASE8_GROUNDING_HEAD_DECISION.md`). The
+evaluation script originally built its own single-box baseline with `argmax_candidate`, while Phase 7
+measured the baseline through `ground_phrase` (threshold box + up to 5 local maxima). Same 16,159
+records, same metric, same cached features — **different decode**: 0.0972 vs 0.0092 (a 10× gap). The
+eval printed `head beats zero-shot: True (+0.1123)` when the matched comparison was +0.0243. The fix
+extracted the decode into one function both paths call
+(`specialists/grounding/inference.py::decode_candidates_from_features`), and re-running the eval
+reproduced the Phase 7 number **exactly** (`|difference| = 0.0000`) — proof the fix is real, not
+cosmetic.
+
+**Status:** `MEASURED`. The trained head beats the zero-shot baseline; the absolute IoU is low, which
+is an honest limitation. See [`EVALUATION.md`](EVALUATION.md) §2.2 and [`MODELS.md`](MODELS.md) §3.4.
+
+---
+
+### 4.3 Optical-SAR fusion — `MEASURED`, ruling `OPEN`
+
+- **Artifact:** `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`.
+- **Tool:** `scripts/eval_fusion_115.py` (read-only) — a **separate, later** step from training.
+- **Split:** held-out **test**, n = **4,000** (`n_scored: 4000`, `split: "test"`).
+- **Label space:** 19 CLC classes; **14 present, 5 absent** in the scored split.
+- **macro-F1 denominator:** all 19 classes (absent classes contribute 0.0) — recorded explicitly.
+
+**Why the metric is computed by a separate tool.** Every `run_record.json` carries
+`pre_registered_metric_computed = false` and `result_status = "PLUMBING_ONLY — fixture/loop evidence,
+NOT a result; the pre-registered 11.5 metric is not computed"`. That flag is **correct about the
+trainer** and must not be "fixed": the trainer fits on `train`/`val` and never opens the test split,
+deliberately, so the held-out split cannot be contaminated by the search over 10 runs, 2 arms and 5
+seeds. The 11.5 metric therefore belongs to a separate, read-only step.
+
+**The tool refuses (exit 2) rather than guessing** when: the head or cache is missing; the split is
+empty; the cache `config_hash` is not `78f1e3700da15aa1` (a different experiment); or the head cannot
+be built against the frozen `(B, 2318)` input contract.
+
+**Measured numbers:**
+
+| Metric | Value | Exact key path |
+|---|---|---|
+| **accuracy** | **0.931** | `accuracy` |
+| **macro F1** | **0.434161** | `macro_f1` |
+| loss | 0.254592 | `loss` |
+| n scored | 4,000 | `n_scored` |
+| num classes | 19 | `num_classes` |
+| macro-F1 denominator | all 19 classes (absent contribute 0.0) | `macro_f1_denominator` |
+| classes present | `[0,2,3,4,5,6,7,8,9,10,12,13,17,18]` | `classes_present` |
+| classes absent | `[1,11,14,15,16]` | `classes_absent` |
+| deciding statistic | **false** | `is_deciding_statistic` |
+| head sha256 | `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab` | `head_sha256` |
+| head bytes | 14,427,457 | `head_bytes` |
+| cache arm | A | `cache_arm` |
+| cache config hash | `78f1e3700da15aa1` | `cache_config_hash` |
 
-## 4. Live validation (behavioural, not accuracy)
+**Optical-SAR accuracy 0.931 must ALWAYS travel with macro-F1 0.434161.** The two are recorded side by
+side in the artifact precisely so neither can be quoted alone.
+
+**Per-class F1 (`_per_class_f1`, 19 terms):**
+
+```
+[1.000, 0.000, 0.428, 0.571, 0.887, 0.000, 0.000, 0.800, 0.929, 0.940,
+ 0.182, 0.000, 0.200, 0.438, 0.000, 0.000, 0.000, 0.877, 0.998]
+```
+
+**Two averages from the same vector — the denominator trap:**
+
+| Averaged over | Value |
+|---|---|
+| **all 19 slots** — the pre-registered definition | **0.434161** |
+| the 14 present classes only | **0.589218** |
+
+A reader who computes the second and compares it to the recorded scalar will wrongly conclude the
+recorded figure is wrong. It is the 19-slot mean by construction. Reproducible:
+`sum(per_class)/19 = 0.43416082670034284`, `sum(per_class[c] for c in present)/14 = 0.5892182648076082`
+(`docs/PHASE12_115_METRIC_COMPUTED.md` §3.5).
+
+**Five absent classes contribute `0.0` by definition — but two PRESENT classes also score `0.0`.**
+Classes **1, 11, 14, 15, 16** are absent; classes **5** and **6** are **present** and score `0.0` — a
+**real total miss**. So part of the low macro-F1 is populational, and part is genuine per-class failure.
+**High accuracy with a wide per-class spread is the signature of prediction dominated by frequent
+classes.**
+
+**Measured test-set label distribution** (`docs/PHASE12_115_METRIC_COMPUTED.md` §3.4):
+
+```
+[13, 0, 63, 39, 846, 4, 1, 6, 453, 246, 4, 0, 9, 18, 0, 0, 0, 34, 2264]
+```
+
+14 of 19 classes present; five have zero samples; top-to-bottom ratio **2,264 : 1** (class 18 vs class
+6). The majority class holds 2,264 / 4,000 = **0.566**; the head scores **0.931**, so the accuracy is
+not a constant predictor. Per-class recall (selected): class 18 n=2264 → **0.997**; class 9 n=246 →
+0.951; class 17 n=34 → 0.941; class 8 n=453 → 0.905; class 4 n=846 → 0.809; class 3 n=39 → 0.410;
+class 12 n=9 → 0.111; classes 5 and 6 → **0.000**.
+
+**Four independent verification checks** (`docs/PHASE12_115_METRIC_COMPUTED.md` §3): (1)
+`evaluate_fusion_head` directly returns `n=4000, accuracy=0.931, macro_f1=0.43416082670034284`; (2)
+rebuilding the input tensor by hand — `concat([optical_gap, sar_gap, joint_gap])` then
+`concat([…, optical_mask, sar_mask])`, widths `(4000,768)×3 + (4000,12) + (4000,2) = 2318` — gives
+0.931 again (this checks feature **ordering**); (3) `sklearn.metrics` `accuracy_score = 0.931`,
+`f1_score(average="macro", labels=range(19), zero_division=0) = 0.43416082670034284`; (4) the per-class
+breakdown above.
+
+**⚠️ The caveat that governs how these numbers may be used.** The cache metadata records
+`label_policy = require_single_label`, `n_skipped_by_policy = 0`. reBEN v2.0 is a **multi-label**
+corpus while the frozen head is a **single-label 19-class softmax**. The extraction used policy (b) —
+restrict to single-label patches — which preserves the frozen architecture exactly but **changes the
+evaluation population**. Single-label patches are **17.57 %** of the corpus (96,537 of 549,488), and the
+rarest class survives as **1 patch** (a 59,204 : 1 imbalance). So this metric **may not** be presented
+as a multi-label BigEarthNet/reBEN result, as comparable to published BigEarthNet numbers, or as a
+statement about all 19 classes. It **may** be presented as the pre-registered 11.5 metric under the
+single-label extraction policy on the held-out split.
+
+**Status:** `MEASURED`, ruling **OPEN**. `is_deciding_statistic: false`; the artifact's `advisory`
+states it *"selects no head, ranks nothing and compares no arms. Whether this constitutes a Phase 12
+pass is the owner's ruling."* No numerical threshold is attached. See [`EVALUATION.md`](EVALUATION.md)
+§2.3 and [`MODELS.md`](MODELS.md) §3.3.
+
+---
+
+### 4.4 Change-VQA — `MEASURED`, ruling `OPEN`
+
+- **Artifact:** `artifacts/change_vqa/run/PROMOTION.json` (`verification` block).
+- **Splits:** `test` (n = **39,686**) and `test2` (n = **31,036**).
+- **Selection:** epoch **8**, chosen on **val answer accuracy 0.700018**, stopped by early stopping
+  (`identity.selected_on`, `identity.stop_reason`).
+- **Answer space:** a **closed 19-answer space** over the eight CDVQA question types; type mask applied
+  by default (`verification.mask_gain: 0.0`).
+
+**Two test sets are reported. Quoting only the better one would be selective.**
+
+| Test set | n | accuracy | macro F1 |
+|---|---|---|---|
+| `test` | 39,686 | **0.697626** | **0.378373** |
+| `test2` | 31,036 | **0.651469** | **0.372309** |
+
+| Field | Value | Exact key path |
+|---|---|---|
+| test accuracy | 0.697626367 | `verification.test_accuracy` |
+| test macro F1 | 0.378373275 | `verification.test_macro_f1` |
+| test2 accuracy | 0.651469262 | `verification.test2_accuracy` |
+| test2 macro F1 | 0.372308516 | `verification.test2_macro_f1` |
+| n scored, test | 39,686 | `verification.n_scored_test` |
+| n scored, test2 | 31,036 | `verification.n_scored_test2` |
+| global majority baseline, test | 0.311546 | `verification.global_majority_baseline_test` |
+| global majority baseline, test2 | 0.178728 | `verification.global_majority_baseline_test2` |
+| mask gain | 0.0 | `verification.mask_gain` |
+| checks passed / failed / unverified | 93 / 0 / 0 | `verification.checks_*` |
+| metric ruling | `OPEN — the plan leaves the accuracy/macro-F1 interpretation owner-gated. No official aggregate metric is asserted here.` | `verification.metric_ruling` |
+
+**The gap between accuracy and macro-F1 means the head is carried by common classes.** The
+global-majority baselines make it concrete: 0.311546 on `test` means a constant predictor scores 0.31
+and the head scores 0.70 — but the macro-F1 of 0.378 shows the per-class picture is far weaker than the
+aggregate.
+
+**Status:** `MEASURED`, ruling **OPEN**. No promotion/acceptance decision has been recorded
+(`PROMOTION.json` → `state.note`). See [`EVALUATION.md`](EVALUATION.md) §2.4 and
+[`MODELS.md`](MODELS.md) §3.2.
+
+---
+
+### 4.5 VLM adapter — `MEASURED`, `ACCEPTANCE-REJECTED`
+
+- **Artifact:** `artifacts/vlm/phase6_closure.json` (`status: CLOSED`).
+- **Split:** a frozen **1,000-question subset** of the test split, whose identity was proven without a
+  model (Gate A″): `available_per_split {val: 6750, test: 7772}`, subset `n = 1000`, test per-class
+  counts an **exact** match to the recovery subset (19 classes, sum 1000).
+
+**The "usable" numbers** (`why_usable_verified.adapted_test`):
+
+| Metric | Value | Exact key path |
+|---|---|---|
+| **exact_match** | **0.963** | `why_usable_verified.adapted_test.exact_match` |
+| **F1** | **0.96432** | `why_usable_verified.adapted_test.f1` |
+| n | 1000 | `…adapted_test.n` |
+| precision | 0.963391 | `…adapted_test.precision` |
+| recall | 0.965251 | `…adapted_test.recall` |
+| tp / fp / tn / fn | 500 / 19 / 463 / 18 | `…adapted_test.confusion` |
+| aggregate test delta | **+49.50 pp** (46.80 → 96.30 pp) | `why_usable_verified.aggregate_test_delta_pp` |
+
+**The "rejected" side** (`why_acceptance_rejected`, rule version **v002**, decision split **test**):
+
+| | |
+|---|---|
+| **V1** | **passes** — test `46.80 → 96.30 pp`, delta **+49.50 pp** (required ≥ +5.00) |
+| **V2** | **fails** — **Mixed forest**: `n=33`, `100.00 → 87.8788 pp`, `drop 12.1212 pp`, `lost_questions 4`, `z 2.1335` |
+
+It fails **both** halves of v002 (`lost ≥ 4` **and** `z ≥ 1.96`). Per item V, a complete run that fails
+V2 is `REJECTED`. The rejection is **narrow**: `n_classes_failed: 1`, `n_classes_held: 5`,
+`n_classes_improved: 11`, `n_classes_total: 19`. The next-worst class (Inland wetlands, −6.6667 pp,
+n=30) lost only 2 questions and sits below V2's materiality floor.
+
+**It is not a split artefact.** The same class also degraded on **val** in Run 1 (drop 6.4516 pp,
+n=31). The adapter hurts Mixed forest on **both** splits. Mixed forest also sits at a 100.00 pp baseline
+on test, so any loss is a drop from the ceiling.
+
+**Residual risk, reported not resolved.** The verdict rests on **4 questions in one class of 33**. With
+no `n ≥ N` floor in V2, a 33-question class can flip the verdict of a run whose aggregate endpoint
+improved by 49.5 pp.
+
+**Both rejection records are preserved** (`preserved_records`): Run 1's manifest `v001` on **val**
+`REJECTED` (3 classes, `val_delta_pp 42.0`, `test_delta_pp null`); the independent test-split
+adjudication `v002` on **test** `REJECTED`; and the recovery manifest `v002` on **val** `ACCEPTED` —
+recorded **for completeness only, NOT final acceptance**, because it decides on the same val subset
+that motivated v002.
+
+**Status:** `MEASURED`; artifact `CLOSED`; headline **`ACCEPTANCE-REJECTED`**. **USABLE ≠ ACCEPTED.**
+The deployed caption/VQA path uses the unadapted model. See [`EVALUATION.md`](EVALUATION.md) §2.5 and
+[`MODELS.md`](MODELS.md) §3.6.
+
+---
+
+### 4.6 Router — `MEASURED`, `TEST NOT RUN`
+
+- **Artifact:** `artifacts/router/threshold_sweep_val.json`.
+- **Split scored:** **validation**, n = **86** (`n_val: 86`, `n_val_examples_scored: 86`,
+  `n_test_examples_scored: 0`, `test_split_touched: false`).
+- **Metric:** overall **ungated** accuracy **0.965116** (`overall_ungated_accuracy`).
+- **Corpus:** **corpus-limited** (`corpus_limited: true`); `corpus_total: 576`, `corpus_groups: 54`;
+  plan minima `plan_min_val_queries: 500`, `plan_min_hard_negatives: 100`; `val_min_support: 8`;
+  `hard_negatives_in_val: 0` (by design — `hn_*` families are held out to TEST).
+- **Sweep:** thresholds 0.50 → 0.99; `select_by: "covered_accuracy"`;
+  `selected.threshold: 0.76` (coverage 0.790698, covered-task accuracy 1.0);
+  `shipped_threshold: 0.70` (coverage 0.848837, covered-task accuracy 0.972603);
+  `delta_vs_shipped: {coverage: -0.0581, covered_task_accuracy: 0.0274}`.
+- **Split sizes:** `{train: 410, test: 80, val: 86}`.
+
+**The artifact's own note, quoted verbatim in substance:** *"corpus-limited: val n=86 vs plan >=500.
+This is NOT a calibration — the corpus is synthetic and too small (min per-class support 8, caption)
+and val carries 0 hard negatives (hn_* families are held out to TEST by design). Selecting a threshold
+here yields a justified default, not a calibrated value. The corpus was NOT padded with generated
+queries. Backlog P1-9's 'n=80' is the TEST split; the sweep target is val n=86. **The test split was
+NOT touched.**"*
+
+**The router TEST split was NOT RUN.** Do not read 0.965116 as a test result. The sweep scored 86
+validation examples and **zero** test examples.
+
+> **Historical record, kept distinct.** `docs/PHASE4_ROUTER_REPORT.md` (verified 2026-09-16, Gate 2
+> PASS) records an **earlier** gate-2 evaluation that included a test column — test task accuracy
+> **0.975**, macro F1 0.976, hard-negative accuracy **0.800**. That evaluation predates the shipped
+> threshold sweep, and its own standing caveat is that *"the 0.975 headline is partly earned on
+> templates the split kept in training. Treat the router as working, not as benchmarked."* The release's
+> position for the shipped artifact is **TEST NOT RUN**; the Phase 4 numbers are retained as a
+> historical record, not promoted to a release benchmark.
+
+**Status:** `MEASURED` — **TEST NOT RUN**. See [`EVALUATION.md`](EVALUATION.md) §2.6 and
+[`MODELS.md`](MODELS.md) §3.5.
+
+---
+
+### 4.7 Calibration — `MEASURED`, negative result
+
+- **Artifact:** `artifacts/calibration_v001.json` (`schema: calibration_v1`).
+- **Fit split:** **val**, n = **16,441** (`temperature_scaling.fitted_on: "Val"`,
+  `temperature_scaling.n_samples: 16441`).
+- **Temperature:** **T = 0.9772731820958189** (`temperature_scaling.temperature`).
+- **Scope:** the R-02 change-VQA head's answer confidence only; other specialists emit their own raw
+  scores and are unaffected (`scope.note`).
+
+| Metric | Before | After | Improvement | Key path |
+|---|---|---|---|---|
+| **ECE** | **0.013755** | **0.014929** | **−0.001174** | `metrics.ece_before` / `metrics.ece_after` / `metrics.ece_improvement` |
+| NLL | 0.689741 | 0.689631 | +0.00011 | `metrics.nll_before` / `metrics.nll_after` |
+
+`ece_improvement = −0.001174` is **negative**: expected calibration error **increased**. NLL improved
+by a negligible 0.00011 (`fit_diagnostics.nll_improvement: 0.0001103574982127542`).
+`metrics.n_bins: 15`, `metrics.n_classes: 19`.
+
+**The scaling is retained because it is part of the frozen configuration, not because it helped.**
+`provenance.held_out_splits_excluded: ["Test", "Test2"]` — the temperature was fitted only on val.
+
+**The reliability diagram is explicitly labelled pre-scaling.** `reliability_diagram.ece` is
+**0.013755** (the *before* value), with the note: *"Equal-width bins over predicted-class confidence.
+ECE is bin-count sensitive and is not an aggregate score."* The diagram on the Benchmark page is
+labelled **pre-scaling** so a reader cannot mistake it for the calibrated result.
+
+**Status:** `MEASURED` — **worse**. A negative result, kept negative. See [`EVALUATION.md`](EVALUATION.md)
+§2.7 and [`MODELS.md`](MODELS.md) §5.
+
+---
+
+## 5. Live validation — behavioural, not accuracy
 
 Accuracy is separate from **behavioural** validation. The deployed stack was driven end-to-end in a
-headed browser, one upload per case, with per-case screenshots and recorded run ids:
+headed browser, one upload per case, with per-case screenshots and recorded run ids.
 
 | Property | Result |
 |---|---|
@@ -84,17 +542,331 @@ headed browser, one upload per case, with per-case screenshots and recorded run
 | Trace fill | **94.4444 %** on every live run |
 | Frontend regression suite | **106 passed** (`tests/unit/test_frontend_live_wiring.py`) |
 
-Every pass produced **fresh run identifiers** — no run id is shared between passes. This is
-behavioural evidence that the pipeline runs and routes correctly; it is **not** an accuracy claim.
+**3 passes × 8 cases = 24 runs, 8/8 each, 0 mock nodes, 94.4444 % trace fill.** Every pass produced
+**fresh run identifiers** — no run id is shared between passes. This is behavioural evidence that the
+pipeline runs and routes correctly; it is **not** an accuracy claim.
+
+### 5.1 Harness integrity — a false-pass bug that was caught
+
+An earlier harness revision typed queries with **synthetic CDP key events**, which Chrome **silently
+drops when the window lacks OS focus**. The harness therefore dispatched the page's *default* query and
+still recorded a "result" — a **false pass**.
+
+The current harness **asserts form state before dispatch**: that the query box really holds the intended
+query, that `#obsTail` reads `ready`, and that both frames are attached for pair tasks. The earlier 8/8
+run was independently checked and confirmed **not** to have been infected (its answers were
+query-specific and the query text was embedded in the answers). This failure mode is recorded because
+it is exactly the kind of silent false-positive an evaluation harness must not have.
+
+### 5.2 Deployed HEADs
+
+The live validation ran against these deployed revisions: Frontend `2d7ae53b482d`,
+Backend `89d80eaddec5`, Inference `5a0936ace491`.
+
+### 5.3 What live validation does and does not establish
 
-## 5. How to reproduce
+- **Establishes:** the pipeline runs end-to-end; the router dispatches the right specialist for the 8
+  cases in each pass; no mock nodes contaminate a live run; the trace is populated to 94.4444 %.
+- **Does not establish:** any accuracy figure. There is no labelled ground truth attached to the 24
+  live runs, so no accuracy can be computed from them.
+
+---
+
+## 6. What is NOT benchmarked
+
+An exhaustive list. Absent numbers are stated, never silently omitted or replaced with an estimate.
+
+| Benchmark | Status | Note |
+|---|---|---|
+| **System-level end-to-end accuracy** | **NOT RUN — none exists** | There is no measured end-to-end benchmark of the full router → specialist → envelope pipeline. No such number is claimed anywhere. |
+| **Router test split** | **NOT RUN** | Only the validation split (n = 86) was scored by the shipped threshold sweep. |
+| **Grounding at 448 for the trained head** | **NOT RUN** | Only the *zero-shot baseline* was measured at 448 (`docs/MODELS.md` §4.1). Re-opening the resolution question for the trained head would be a new pre-registered experiment. |
+| **Benchmark adapters** | **NOT RUN** | Adapter-based benchmark runs were not executed. |
+| **Efficiency / latency benchmark** | **NOT RUN (system-level)** | Per-specialist latency is recorded incidentally in artifacts (e.g. grounding `latency_ms_per_image` 2.205 ms for the threshold decode, 0.655 ms for argmax; CROMA forward pass 0.89 s for a batch of 2 at 120 px), but there is **no end-to-end latency benchmark**. |
+| **Cross-dataset generalisation** | **NOT RUN** | Each specialist is evaluated only on its own training-family test split. |
+| **Human evaluation** | **NOT RUN** | No human rating of answers was performed. |
+| **Robustness / adversarial evaluation** | **NOT RUN** | No perturbation, adversarial, or out-of-distribution robustness study was performed. |
+| **Multi-label BigEarthNet / reBEN result** | **NOT PRODUCED** | The local subset is 100 % single-label against the official 1–11 multi-label scheme. Metrics are **not comparable** to published multi-label numbers. |
+| **Grounding head wired and benchmarked in the serving path** | **NOT RUN** | `specialists/grounding/inference.py` exposes only the zero-shot path as of the Phase 8 record; wiring the head is the remaining integration step. |
+| **Change-VQA served end-to-end** | **NOT RUN** | Serving needs both a trained head and a trained feature extractor; neither is wired by default, and an untrained head refuses to answer. |
+| **Optical-SAR multi-label evaluation** | **NOT RUN** | See the single-label caveat (§4.3). |
+| **Change-VQA on a third held-out set** | **NOT RUN** | Only `test` and `test2` exist. |
+| **Any private / hidden evaluation data** | **NOT ACCESSED** | `evaluation.hidden_data_access: false` in `configs/base.yaml`. |
+| **Calibrated reliability curve** | **NOT PLOTTED** | Only the pre-scaling diagram exists; the calibrated curve is not plotted. |
+| **Per-class support for the router test split** | **NOT RUN** | The test split was not scored, so it has no support table. |
+
+---
+
+## 7. How to reproduce
+
+### 7.1 Verify every number in this document against its artifact
 
 ```bash
-# verify every README/benchmark number against its artifact
+# verify every README/benchmark number against its source artifact
 python release/tools/verify_readme_metrics.py
 ```
 
-The script resolves nested artifact keys (including keys that themselves contain dots, such as the
-`recall` dict keyed `"0.10"/"0.25"/"0.50"`) and compares each value at the precision printed in the
-README. It exits non-zero if any claim fails. See [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) for the
-full reproduction guide.
+The script resolves nested artifact keys — including keys that themselves contain dots, such as the
+`recall` dict keyed `"0.10"/"0.25"/"0.50"` — and compares each value at the precision printed in the
+README. It exits **non-zero if any claim fails**, and prints `ALL CLAIMS VERIFIED` (exit 0) only when
+all 20 numeric claims match and the status assertions hold. Committed output:
+[`../tools/readme_metrics_report.txt`](../tools/readme_metrics_report.txt).
+
+### 7.2 Reproduce the individual metrics
+
+| Metric | Reproduction path |
+|---|---|
+| change | `scripts/eval_change.py` (refuses to score on a config-hash drift, exit 3) |
+| grounding | `scripts/exp_grounding_resolution.py` + `scripts/analyze_grounding_resolution.py`; the two eval JSONs are recomputable from the per-sample JSONL without re-running the encoder |
+| optical-SAR | `python scripts/eval_fusion_115.py --head … --cache-dir artifacts/optical_sar/fusion_features --split test --report …` (exit 0 = computed; exit 2 = refused) |
+| change-VQA | `scripts/evaluate_change_vqa.py` (default checkpoint `artifacts/change_vqa/run/head.pt`) |
+| VLM | `scripts/phase6_close.py` regenerates the closure record from the evidence; `scripts/phase6_adjudicate_test.py` re-runs the independent test-split adjudication |
+| calibration | `load_calibration(config, base_dir='configs')` consumes `calibration_v001.json` |
+| router | `scripts/train_router.py` (cached-embedding training); the sweep artifact is `threshold_sweep_val.json` |
+
+See [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) for the full reproduction guide and
+[`EVALUATION.md`](EVALUATION.md) for how each number was produced.
+
+### 7.3 Reproduce the live validation
+
+The 24 live runs were driven in a headed browser with per-case screenshots and recorded run ids. The
+harness asserts form state before dispatch (§5.1). See [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) for
+the harness entry point.
+
+### 7.4 Verify the released artifacts
+
+```bash
+sha256sum -c models/checksums.sha256
+```
+
+---
+
+## 8. The verification tool — exactly what it checks
+
+`release/tools/verify_readme_metrics.py` is read-only: it writes nothing and prints a report suitable
+for pasting into a release-verification record. It holds a hard-coded `CLAIMS` list of
+`(label, artifact path, dotted key, printed value)` tuples. Key names were **discovered by walking the
+artifacts, not assumed** — several live under nested paths (change metrics are `metrics.pooled.iou`;
+grounding is `results.head_threshold.mean_best_iou`; VLM is `why_usable_verified.adapted_test.*`).
+
+### 8.1 The 20 numeric claims
+
+| # | Claim | Artifact | Key path | Printed |
+|---|---|---|---|---|
+| 1 | change pooled IoU | `artifacts/change/eval_test/eval_result.json` | `metrics.pooled.iou` | 0.8122 |
+| 2 | change macro IoU | `…/eval_result.json` | `metrics.macro.miou` | 0.8457 |
+| 3 | change pooled F1 | `…/eval_result.json` | `metrics.pooled.f1` | 0.8964 |
+| 4 | grounding canonical head_threshold mean_best_IoU | `…/eval_result_canonical.json` | `results.head_threshold.mean_best_iou` | 0.2838 |
+| 5 | grounding canonical head_threshold recall@0.5 | `…/eval_result_canonical.json` | `results.head_threshold.recall.0.50` | 0.2198 |
+| 6 | grounding matched6 head_threshold mean_best_IoU | `…/eval_result_matched6.json` | `results.head_threshold.mean_best_iou` | 0.2566 |
+| 7 | grounding matched6 head_threshold recall@0.5 | `…/eval_result_matched6.json` | `results.head_threshold.recall.0.50` | 0.1938 |
+| 8 | grounding head_argmax mean_best_IoU (canonical) | `…/eval_result_canonical.json` | `results.head_argmax.mean_best_iou` | 0.1215 |
+| 9 | grounding zero-shot baseline IoU (canonical) | `…/eval_result_canonical.json` | `results.zero_shot_matched.mean_best_iou` | 0.0972 |
+| 10 | optical-SAR fusion accuracy | `…/pre_registered_115_metric.json` | `accuracy` | 0.931 |
+| 11 | optical-SAR fusion macro_F1 | `…/pre_registered_115_metric.json` | `macro_f1` | 0.434161 |
+| 12 | change_vqa test accuracy | `artifacts/change_vqa/run/PROMOTION.json` | `verification.test_accuracy` | 0.697626 |
+| 13 | change_vqa test macro_F1 | `…/PROMOTION.json` | `verification.test_macro_f1` | 0.378373 |
+| 14 | change_vqa test2 accuracy | `…/PROMOTION.json` | `verification.test2_accuracy` | 0.651469 |
+| 15 | change_vqa test2 macro_F1 | `…/PROMOTION.json` | `verification.test2_macro_f1` | 0.372309 |
+| 16 | router overall ungated accuracy | `artifacts/router/threshold_sweep_val.json` | `overall_ungated_accuracy` | 0.965116 |
+| 17 | calibration ECE before scaling | `artifacts/calibration_v001.json` | `metrics.ece_before` | 0.013755 |
+| 18 | calibration ECE after scaling | `artifacts/calibration_v001.json` | `metrics.ece_after` | 0.014929 |
+| 19 | VLM adapter exact_match | `artifacts/vlm/phase6_closure.json` | `why_usable_verified.adapted_test.exact_match` | 0.963 |
+| 20 | VLM adapter F1 | `artifacts/vlm/phase6_closure.json` | `why_usable_verified.adapted_test.f1` | 0.96432 |
+
+### 8.2 The status assertions
+
+Beyond the 20 numeric claims, the tool asserts artifact-declared **statuses**:
+
+```
+VLM headline contains ACCEPTANCE-REJECTED : True
+VLM status                               : CLOSED
+router corpus_limited                    : True
+router n_val                             : 86
+calibration temperature                  : 0.9772731820958189
+calibration ece_improvement              : -0.001174  (negative => calibration did NOT help)
+```
+
+### 8.3 The committed output
+
+`release/tools/readme_metrics_report.txt` records all 20 rows as `MATCH`, all six status assertions as
+expected, and the final line:
+
+```
+RESULT: ALL CLAIMS VERIFIED
+```
+
+The tool exits `0` on that result and `1` otherwise, so a drifted artifact fails the release check
+rather than passing silently.
+
+### 8.4 What the tool does not check
+
+- It does **not** check the `n` or `split` fields. A value could match while the population changed;
+  the `n`/`split` columns in §1 are therefore maintained by hand against the artifacts.
+- It does **not** check the calibration reliability diagram, the grounding per-sample JSONL, or the
+  VLM per-class tables.
+- It does **not** verify the live-validation record (§5), which is behavioural and not artifact-scored.
+
+---
+
+## 9. Artifact schemas and key-path resolution
+
+Every metric in §1 is addressed by a dotted **key path** into a JSON artifact. Two mechanics matter,
+because getting either wrong produces a `NOKEY` failure that looks like a missing metric rather than a
+bug in the reader.
+
+### 9.1 The dotted-key problem
+
+Some artifact keys themselves contain dots. The grounding artifacts key their recall dict as
+`{"0.10": …, "0.25": …, "0.50": …}`. A naive `dotted.split(".")` walk would break
+`results.head_threshold.recall.0.50` into `… recall → 0 → 50` and fail — even though the value exists.
+
+`verify_readme_metrics.py` resolves this by **preferring the longest matching key at each step**: at a
+dict node it tries the longest possible joined key first, and only shortens when no key matches. That
+is why the recall claim is written `results.head_threshold.recall.0.50` and resolves to the `"0.50"`
+entry rather than to a nested `0` object.
+
+### 9.2 Top-level schema of each artifact family
+
+These are the top-level keys actually present in each artifact, read from the files:
+
+| Artifact | Top-level keys (abridged to the load-bearing ones) |
+|---|---|
+| `change/eval_test/eval_result.json` | `artifact`, `checkpoint`, `checkpoint_config_hash`, `checkpoint_config_hash_checked`, `checkpoint_embedded_config`, `config_drift`, `config_hash`, `created_at`, `device`, `environment`, `metrics`, `n`, `n_images_with_change`, `change_fraction_quantiles`, `per_image_change_fraction` |
+| `grounding/…/eval_result_canonical.json` | `checkpoint_config_hash`, `config_drift`, `config_hash`, `created_at`, `device`, `environment`, `eval_cache`, `frozen_config_evaluation`, `grid`, `head_decode`, `limited_run`, `n_eval_records`, `phase7_reference`, `resolution`, `results` |
+| `grounding/…/eval_result_matched6.json` | same shape; `head_decode.top_k` differs (6 vs 20) |
+| `optical_sar/…/pre_registered_115_metric.json` | `tool`, `metric`, `definition`, `split`, `head_path`, `head_sha256`, `head_bytes`, `cache_path`, `cache_arm`, `cache_config_hash`, `n_scored`, `accuracy`, `macro_f1`, `loss`, `num_classes`, `macro_f1_denominator`, `classes_present`, `classes_absent`, `_per_class_f1`, `is_deciding_statistic`, `advisory` |
+| `change_vqa/run/PROMOTION.json` | `schema`, `promoted_utc`, `promoted_by`, `artifact`, `source`, `identity`, `frozen_dependency`, `verification`, `state`, `serving_wiring` |
+| `router/threshold_sweep_val.json` | `adapter_config_hash`, `adapter_encoder`, `adapter_path`, `artifact`, `config_hash`, `corpus_groups`, `corpus_limited`, `corpus_total`, `created_at`, `delta_vs_shipped`, `device`, `encoder_type`, `environment`, `force`, `hard_negatives_in_val`, `n_test_examples_scored`, `n_val`, `n_val_examples_scored`, `note`, `overall_ungated_accuracy`, `plan_min_hard_negatives`, `plan_min_val_queries`, `rows`, `seconds`, `select_by`, `selected`, `shipped_row`, `shipped_threshold`, `split`, `split_sizes`, `test_split_touched`, `thresholds`, `val_min_support`, `val_task_counts` |
+| `calibration_v001.json` | `consumer_contract`, `created_utc`, `fit_diagnostics`, `metrics`, `provenance`, `reliability_diagram`, `schema`, `scope`, `temperature_scaling`, `type_mask_applied` |
+| `vlm/phase6_closure.json` | `artifact_verification`, `evidence_index`, `forward_rule`, `generated_at`, `headline`, `kind`, `not_a_retrain`, `phase`, `preserved_records`, `production_adapter`, `rule_unchanged_by_closure`, `status`, `verified_vs_accepted`, `what_closure_does_not_claim`, `why_acceptance_rejected`, `why_usable_verified` |
+
+### 9.3 Status and provenance fields carried by every artifact
+
+The artifacts are not just numbers; each carries the fields that make its number auditable:
+
+| Field | Meaning | Where |
+|---|---|---|
+| `config_hash` | the frozen registry hash the run was made against (`78f1e3700da15aa1`) | change, grounding, optical-SAR, change-VQA, router, calibration, VLM |
+| `checkpoint_config_hash` | the config hash **baked into the weights** | change, grounding |
+| `config_drift` | whether the two hashes disagreed at evaluation time | change, grounding (both `false`) |
+| `created_at` / `generated_at` | when the artifact was produced | all |
+| `device` / `environment` | device and library versions | change, grounding, router |
+| `split` | the population scored | optical-SAR (`test`), router (`val`) |
+| `n` / `n_scored` / `n_eval_records` | the population size | all |
+
+**Why `config_drift: false` matters.** The evaluation is only valid against the configuration the
+weights were trained under. `checkpoint_config_hash_checked: true` and `config_drift: false` together
+record that the check was *performed* and *passed* — not merely omitted.
+
+### 9.4 The `is_deciding_statistic` and ruling fields
+
+Two artifacts carry an explicit statement that their number is **not** a decision:
+
+- optical-SAR: `is_deciding_statistic: false`, with an `advisory` that the tool *"selects no head,
+  ranks nothing and compares no arms."*
+- change-VQA: `verification.metric_ruling: "OPEN — … No official aggregate metric is asserted here."`
+
+These fields exist so that a later reader cannot promote a measurement into a ruling by quoting it out
+of context. Where a ruling exists, it is recorded separately (e.g. the optical-SAR A/B arm ruling was
+made on `best_val_accuracy` — A 0.837100 vs B 0.839100, floor 0.0285 → **Arm A retained**, ruling
+R-14 — not on the 0.931/0.434161 held-out metric).
+
+---
+
+## 10. Metric glossary and aggregation arithmetic
+
+Definitions, so that a reader can recompute every number from the stored confusion counts or per-class
+vectors.
+
+### 10.1 Pooled vs macro
+
+| Term | Definition | Where used |
+|---|---|---|
+| **pooled IoU** | IoU over the union of all pixels, computed from the global `tp/fp/fn` counts | change (`metrics.pooled.iou` = 0.8122) |
+| **macro IoU** | unweighted mean of per-class IoU. For binary change this is the mean over {change, no-change} | change (`metrics.macro.miou` = 0.8457) |
+| **pooled F1** | `2·tp / (2·tp + fp + fn)` from the global counts | change (`metrics.pooled.f1` = 0.8964) |
+| **macro F1** | unweighted mean of per-class F1 | change (`metrics.macro.f1` = 0.7962); optical-SAR (`macro_f1` = 0.434161) |
+
+For the change artifact, `metrics.macro.iou` = 0.718 is the **change-class** IoU, while
+`metrics.macro.miou` = 0.8457 is the **mean over both classes** — these are different numbers from the
+same block, and the headline uses `miou`.
+
+### 10.2 Mean best IoU, mean matched IoU, recall@IoU
+
+| Term | Definition | Where used |
+|---|---|---|
+| **mean best IoU** | for each record, the maximum IoU over all predicted boxes; then the mean over records | grounding (`mean_best_iou` = 0.2838 / 0.2566 / 0.1215 / 0.0972) |
+| **mean matched IoU** | the IoU of the *matched* prediction only, so unmatched records contribute 0 | grounding (`mean_matched_iou`, equal to `mean_best_iou` in all three strategies) |
+| **recall@0.10 / 0.25 / 0.50** | fraction of records whose best predicted box reaches that IoU | grounding (`results.*.recall`) |
+
+**Mean best IoU is a max over predictions, so emitting more boxes raises it mechanically.** That is
+exactly why the candidate count is recorded (`head_decode.top_k`) and why the decode-matched protocol
+(top_k 6 vs the baseline's 5.99) is reported alongside the config default (top_k 20). See §4.2.
+
+### 10.3 The macro-F1 denominator
+
+For a K-class head, `macro_f1 = (1/K) · Σ_c F1_c`. The **denominator is the label space**, not the
+number of classes present in the scored split.
+
+- optical-SAR: `K = 19` (`num_classes`), 14 present, 5 absent. Absent classes contribute `0.0`.
+  All-19 mean = **0.434161**; present-14 mean = **0.589218**.
+- The artifact records `macro_f1_denominator: "all 19 classes (absent classes contribute 0.0)"`,
+  `classes_present`, `classes_absent`, and the full 19-term `_per_class_f1` vector so the denominator
+  is auditable rather than assumed.
+
+### 10.4 Percentage points vs fractions
+
+Some artifacts record a **fraction** (0.963) and some a **percentage-point delta** (49.5 pp). The VLM
+record uses both: `exact_match: 0.963` is a fraction, while `aggregate_test_delta_pp: 49.5` and the
+baseline/adapted `46.80 → 96.30 pp` are percentage points. A reader must not confuse `+49.5 pp` with
+`+0.495` of a fraction — the former is a 49.5-point move on a 0–100 scale.
+
+### 10.5 ECE and NLL
+
+| Term | Definition | Where used |
+|---|---|---|
+| **ECE** | expected calibration error: bin-count-sensitive average gap between confidence and accuracy over equal-width bins | calibration (`metrics.ece_before` = 0.013755, `metrics.ece_after` = 0.014929, `n_bins` = 15) |
+| **NLL** | mean negative log-likelihood | calibration (`metrics.nll_before` = 0.689741, `metrics.nll_after` = 0.689631) |
+
+**ECE is bin-count sensitive and is not an aggregate score** — recorded verbatim in the artifact's
+`reliability_diagram.note`. A smaller ECE is better; here it **increased** (`ece_improvement` =
+−0.001174), which is a negative result.
+
+### 10.6 Coverage, fallback rate and ungated accuracy (router)
+
+| Term | Definition | Where used |
+|---|---|---|
+| **overall ungated accuracy** | task accuracy over all examples, whether or not the confidence gate fired | router (`overall_ungated_accuracy` = 0.965116) |
+| **coverage** | fraction of examples the gate accepted | router (`rows[*].coverage`; shipped 0.848837 at threshold 0.70) |
+| **covered task accuracy** | task accuracy **among gated-in examples only** | router (`rows[*].covered_task_accuracy`; shipped 0.972603 at 0.70) |
+| **fallback rate** | fraction routed to the deterministic lexical fallback | router (`rows[*].fallback_rate`; shipped 0.151163 at 0.70) |
+
+The headline is the **ungated** figure, because a gated accuracy without its coverage is not comparable
+across thresholds. That is why the sweep reports both, and why "ungated" is part of the label.
+
+### 10.7 Why the router's `selected` threshold differs from the `shipped` threshold
+
+`select_by: "covered_accuracy"` maximises accuracy among gated-in examples, which is monotone in the
+threshold (at 0.76 the covered accuracy is 1.0 on 68 examples). The **shipped** threshold is 0.70
+(coverage 0.848837, covered accuracy 0.972603), which is the config default. The `delta_vs_shipped`
+block records the difference: `{coverage: -0.0581, covered_task_accuracy: 0.0274}` — i.e. the selected
+threshold covers 5.81 points fewer examples for 2.74 points more covered accuracy. The artifact is
+explicit that this is *"a justified default, not a calibrated value."*
+
+---
+
+## 11. Evidence index
+
+| Topic | Evidence |
+|---|---|
+| Verification tool | `release/tools/verify_readme_metrics.py`; committed output `release/tools/readme_metrics_report.txt` |
+| Change | `artifacts/change/eval_test/eval_result.json` |
+| Grounding (canonical / matched6) | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json`, `…/eval_result_matched6.json`; `docs/PHASE7_RESOLUTION_DECISION.md`, `docs/PHASE8_GROUNDING_HEAD_DECISION.md` |
+| Optical-SAR | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`; `docs/PHASE12_115_METRIC_COMPUTED.md`, `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` |
+| Change-VQA | `artifacts/change_vqa/run/PROMOTION.json` |
+| Router | `artifacts/router/threshold_sweep_val.json`; `docs/PHASE4_ROUTER_REPORT.md` |
+| Calibration | `artifacts/calibration_v001.json` |
+| VLM | `artifacts/vlm/phase6_closure.json`; `docs/PHASE6_CLOSURE.md` |
+| Live validation | the live-validation record and per-case screenshots; `tests/unit/test_frontend_live_wiring.py` (106 passed) |
+| Cross-links | [`EVALUATION.md`](EVALUATION.md) (how each number was produced), [`MODELS.md`](MODELS.md) (the artifacts behind each number), [`DATASETS.md`](DATASETS.md) (the corpora), [`TRAINING.md`](TRAINING.md) (how each head was trained), [`LIMITATIONS.md`](LIMITATIONS.md) (the honest catalogue of gaps), [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) (reproduction guide) |