docs/BENCHMARKS.md

Benchmarks — deep reference

Every number here is artifact-backed. Where a metric exists, its source file and the exact key path inside that file are named. Where a benchmark does not exist, that is stated explicitly and tagged NOT RUN — an absent number is never silently omitted, never estimated, and never replaced with a plausible-looking substitute.

Status tags: VERIFIED · MEASURED · NOT RUN · OPEN · REJECTED · BLOCKED.

The single most important rule in this document: do not fabricate. If a value is not in an artifact that was read, it does not appear here. Where a fact is not established, this document writes UNKNOWN — not established from the available evidence.

There is no system-level end-to-end benchmark. The router → specialist → envelope pipeline has never been scored end-to-end, and no end-to-end accuracy is claimed anywhere in this repository. What exists is (a) per-specialist metrics on their own training-family test splits, and (b) a behavioural live-validation record (§5) that proves the pipeline runs and routes, not that it is accurate.

---

Table of contents

1. Headline table
2. Source artifacts
3. The honesty rules this document follows
4. Per-task evaluation protocols
   • 4.1 Change detection
   • 4.2 Grounding
   • 4.3 Optical-SAR fusion
   • 4.4 Change-VQA
   • 4.5 VLM adapter
   • 4.6 Router
   • 4.7 Calibration
5. Live validation — behavioural, not accuracy
6. What is NOT benchmarked
7. How to reproduce
8. The verification tool — exactly what it checks
9. Artifact schemas and key-path resolution
10. Metric glossary and aggregation arithmetic
11. Evidence index

---

1. Headline table

Every row names its source artifact and the exact key path within it. The n and split columns are part of the claim, not decoration: a metric without its population is not a result.

Capability	Metric	Value	Split / protocol	n	Source artifact → key path	Status
Change detection	pooled IoU	0.8122	LEVIR-CD-256 test, thr 0.50	2,048	artifacts/change/eval_test/eval_result.json → metrics.pooled.iou	VERIFIED
Change detection	macro IoU	0.8457	same	2,048	…/eval_result.json → metrics.macro.miou	VERIFIED
Change detection	pooled F1	0.8964	same	2,048	…/eval_result.json → metrics.pooled.f1	VERIFIED
Grounding	mean best IoU	0.2838	VRSBench, canonical (head threshold, top_k 20)	16,159	…/eval_result_canonical.json → results.head_threshold.mean_best_iou	MEASURED (2 protocols)
Grounding	recall@0.5	0.2198	canonical	16,159	…/eval_result_canonical.json → results.head_threshold.recall.0.50	MEASURED (2 protocols)
Grounding	mean best IoU	0.2566	VRSBench, matched6 (head threshold, top_k 6)	16,159	…/eval_result_matched6.json → results.head_threshold.mean_best_iou	MEASURED (2 protocols)
Grounding	recall@0.5	0.1938	matched6	16,159	…/eval_result_matched6.json → results.head_threshold.recall.0.50	MEASURED (2 protocols)
Grounding	head-argmax IoU	0.1215	canonical (argmax decode)	16,159	…/eval_result_canonical.json → results.head_argmax.mean_best_iou	MEASURED
Grounding	zero-shot baseline IoU	0.0972	canonical (no trained head)	16,159	…/eval_result_canonical.json → results.zero_shot_matched.mean_best_iou	MEASURED (baseline)
Optical-SAR fusion	accuracy	0.931	held-out test, 19 classes	4,000	…/fusion_head_production_v001/pre_registered_115_metric.json → accuracy	MEASURED, ruling OPEN
Optical-SAR fusion	macro F1	0.434161	same	4,000	…/pre_registered_115_metric.json → macro_f1	MEASURED, ruling OPEN
Change-VQA	accuracy	0.697626	test	39,686	artifacts/change_vqa/run/PROMOTION.json → verification.test_accuracy	MEASURED, ruling OPEN
Change-VQA	macro F1	0.378373	test	39,686	…/PROMOTION.json → verification.test_macro_f1	MEASURED, ruling OPEN
Change-VQA	accuracy (2nd set)	0.651469	test2	31,036	…/PROMOTION.json → verification.test2_accuracy	MEASURED, ruling OPEN
Change-VQA	macro F1 (2nd set)	0.372309	test2	31,036	…/PROMOTION.json → verification.test2_macro_f1	MEASURED, ruling OPEN
VLM (adapted)	exact_match	0.963	frozen 1,000-question subset	1,000	artifacts/vlm/phase6_closure.json → why_usable_verified.adapted_test.exact_match	MEASURED, ACCEPTANCE-REJECTED
VLM (adapted)	F1	0.96432	same	1,000	…/phase6_closure.json → why_usable_verified.adapted_test.f1	MEASURED, ACCEPTANCE-REJECTED
Router	overall ungated accuracy	0.965116	val, corpus-limited	86	artifacts/router/threshold_sweep_val.json → overall_ungated_accuracy	MEASURED — TEST NOT RUN
Calibration	ECE before / after	0.013755 → 0.014929	val, T = 0.9773	16,441	artifacts/calibration_v001.json → metrics.ece_before / metrics.ece_after	MEASURED — worse

1.1 The three facts most likely to be misread

1. Grounding is reported under two protocols and two decode variants. The trained head scores 0.2838 (canonical) and 0.2566 (matched6) mean best IoU; the head-argmax decode scores 0.1215 and the zero-shot baseline 0.0972. Quoting one cell of that grid alone is selective. The matched6 protocol is the decode-matched comparison (head emits 6 boxes, baseline emits 5.99); the canonical protocol is the config default (top_k 20).
2. Change-VQA is reported on two test sets. test (0.697626 / 0.378373) and test2 (0.651469 / 0.372309). Quoting only the better one would be selective.
3. Router 0.965116 is validation, ungated, n = 86. It is not a test result; the router test split was NOT RUN for the shipped artifact.

1.2 The only VERIFIED headline

The change-detection triple — pooled IoU 0.8122 / macro IoU 0.8457 / pooled F1 0.8964 — is the only headline in this repository that carries the VERIFIED tag, because it is the only one measured against a single, immutable public test split with a frozen threshold. Everything else is MEASURED (real, artifact-backed, but with an open ruling, a protocol caveat, or a validation-only population).

---

2. Source artifacts

Metric family	Artifact (paths relative to the source repository root)
change	artifacts/change/eval_test/eval_result.json
grounding (canonical)	artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json
grounding (matched6)	artifacts/grounding/remoteclip_grounding_v001/eval_result_matched6.json
optical-SAR	artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json
change-VQA	artifacts/change_vqa/run/PROMOTION.json
router	artifacts/router/threshold_sweep_val.json
calibration	artifacts/calibration_v001.json
VLM	artifacts/vlm/phase6_closure.json

Each of these artifacts carries the frozen config hash 78f1e3700da15aa1 (where it records a hash at all), which ties it to the same configs/base.yaml. A config edit moves the hash and detaches the number from its configuration — which is why the change head is wired into serving via a registry override rather than a config key (docs/MODELS.md §3.1).

All 20 numeric claims in the headline table are checked against these files by ../tools/verify_readme_metrics.py. Its output (ALL CLAIMS VERIFIED) is committed as ../tools/readme_metrics_report.txt. §8 documents exactly what that tool checks.

---

3. The honesty rules this document follows

These are not aspirations. Each one is a rule that changed what is reported, and each is checkable against the artifacts.

1. Evidence before claims. Every reported number has an artifact path. A number with no artifact does not appear in this document.
2. Two protocols are never collapsed. Grounding is reported under both the canonical and matched6 protocols. Quoting 0.2838 alone would be selective; quoting 0.2566 alone would be too.
3. Two decode variants are never collapsed into the trained-head claim. head_argmax (0.1215) is not apples-to-apples with the multi-box baseline (mean best IoU is a max over predictions, so 1 box vs ~6 boxes flatters the head). It is reported because it is the number comparable to the Phase 7 resolution experiment's zero-shot argmax — not because it decides anything.
4. Two test sets are never collapsed. Change-VQA is reported on test and test2.
5. accuracy never travels without macro-F1. For imbalanced multi-class heads (optical-SAR, change-VQA) the macro-F1 is reported alongside accuracy, always, in the same table, with the same split and the same n.
6. Validation is not test. The router figure is labelled "overall ungated accuracy", val, n = 86, corpus-limited. It is not a test result.
7. A negative result stays negative. Calibration ECE worsened (0.013755 → 0.014929) and is shown worsening. It is retained because it is in the frozen config, not because it helped.
8. USABLE ≠ ACCEPTED. The VLM adapter's metrics are real and reproducible; the artifact is nevertheless ACCEPTANCE-REJECTED. Both statements appear together.
9. A metric is never promoted to a system-level claim. There is no end-to-end benchmark, and none is invented by averaging the per-task numbers.
10. An absent number is stated, not omitted. §6 lists everything that is NOT RUN.
11. A protocol-sensitive number carries its protocol. Grounding always names canonical vs matched6 and the decode variant; optical-SAR always names the single-label subset and the 19-slot denominator.
12. A pre-registered rule is applied as written. The grounding resolution decision was decided by a rule fixed before the result was seen, and the artifact records rule_changed_since_preregistration: false (docs/MODELS.md §4.1).
13. A metric with a definitional detail states it. The optical-SAR macro-F1 of 0.434161 is the mean over all 19 class slots, not the 14 present classes — averaging over the present classes gives 0.589218, and the artifact records macro_f1_denominator, classes_present, classes_absent and the full 19-term vector so the difference cannot be misread.

---

4. Per-task evaluation protocols

4.1 Change detection — VERIFIED

• Artifact: artifacts/change/eval_test/eval_result.json.
• Split: LEVIR-CD-256 test, n = 2,048 (immutable public split).
• Threshold: 0.50, frozen in configs/base.yaml (change.threshold).
• Metrics: pooled IoU / macro IoU / pooled F1, plus the full confusion counts so any metric can be recomputed rather than trusted.
• Protocol detail: tiles of 256 px (change.tile_size: 256), overlap 0 (change.tile_overlap: 0), minimum component 32 px (change.min_component_pixels: 32).

Why both pooled and macro. The test split is only ≈ 5 % changed pixels (metrics.mean_change_fraction: 0.0509), and the median tile has zero changed pixels (change_fraction_quantiles.p50: 0.0). Pooled IoU (0.8122) and macro IoU (0.8457) answer different questions about that imbalance.

Measured numbers (all key paths under metrics):

Metric	Value	Key path
pooled IoU	0.8122	pooled.iou
pooled F1	0.8964	pooled.f1
pooled mean IoU (miou)	0.9007	pooled.miou
pooled precision	0.9195	pooled.precision
pooled recall	0.8745	pooled.recall
macro IoU	0.8457	macro.miou
macro F1	0.7962	macro.f1
macro IoU (iou, change-class IoU)	0.718	macro.iou
macro precision	0.8506	macro.precision
macro recall	0.7757	macro.recall
tp	5,978,997	pooled.tp
fp	523,658	pooled.fp
fn	858,407	pooled.fn
tn	126,856,666	pooled.tn
n_pixels	134,217,728	pooled.n_pixels

Population: n: 2048, n_images_with_change: 935, change_fraction_quantiles: {min: 0.0, p50: 0.0, p90: 0.197205, max: 0.684937}.

Environment and integrity: device cuda, torch 2.10.0+cu128, Python 3.12.13, seconds: 55.359; config_hash: 78f1e3700da15aa1, checkpoint_config_hash: 78f1e3700da15aa1, checkpoint_config_hash_checked: true, config_drift: false.

Status: VERIFIED. See EVALUATION.md §2.1 and MODELS.md §3.1.

---

4.2 Grounding — MEASURED, two protocols × two decode variants

• Artifacts: artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json (head_decode.top_k: 20) and …/eval_result_matched6.json (head_decode.top_k: 6).
• Split: VRSBench eval split, n = 16,159 records, scored once, no sampling (n_eval_records: 16159, limited_run: false).
• Resolution: 224 (resolution: 224, grid: 7), frozen; 448 rejected by a pre-registered paired test (docs/MODELS.md §4.1).
• Box convention: VRSBench stores boxes normalised to 0–100; this project stores 0–1, via the declared grounding.benchmark_box_scale: 100.0 so the conversion cannot be applied twice.
• Protocols: canonical and matched6 — both reported.
• Decode variants: head threshold (the shipped decode), head argmax, and zero-shot matched (the baseline).

The head_decode block is recorded in each artifact — config_default_top_k: 20, nms_iou: 0.5, score_threshold: 0.4, top_k: 20 (canonical) / top_k: 6 (matched6). Without it the artifact could not say which setting produced its number, and a re-run at the config default would silently yield a different figure.

The full grid — canonical (eval_result_canonical.json → results):

Strategy	mean_best_iou	recall.0.10	recall.0.25	recall.0.50	latency_ms_per_image	seconds
head_argmax	0.1215	0.3183	0.2088	0.0795	0.655	10.6
head_threshold (top_k 20)	0.2838	0.6882	0.5047	0.2198	2.205	35.6
zero_shot_matched	0.0972	0.3298	0.1188	0.0234	—	17.9

The full grid — matched6 (eval_result_matched6.json → results):

Strategy	mean_best_iou	recall.0.10	recall.0.25	recall.0.50	latency_ms_per_image	seconds
head_argmax	0.1215	0.3183	0.2088	0.0795	0.652	10.5
head_threshold (top_k 6)	0.2566	0.6315	0.4545	0.1938	2.158	34.9
zero_shot_matched	0.0972	0.3298	0.1188	0.0234	—	15.5

Both artifacts carry phase7_reference: {mean_best_iou: 0.0972, recall_at_0.50: 0.0234, source: "docs/PHASE7_RESOLUTION_DECISION.md"} — the recorded baseline against which the head is judged.

Decode-matched delta (head threshold top_k = 6 vs zero-shot): +0.1594 IoU, +0.1704 Recall@0.50. The pre-registered bar was MIN_IMPROVEMENT_IOU = 0.02; the measured margin is 8× the bar.

Why head_argmax is not the trained-head claim. 1 box against ~6 boxes flatters the head, because mean best IoU takes the max over predictions. It is reported for comparability with the Phase 7 resolution experiment, not as the head's result.

A decode defect this protocol had to fix first (docs/PHASE8_GROUNDING_HEAD_DECISION.md). The evaluation script originally built its own single-box baseline with argmax_candidate, while Phase 7 measured the baseline through ground_phrase (threshold box + up to 5 local maxima). Same 16,159 records, same metric, same cached features — different decode: 0.0972 vs 0.0092 (a 10× gap). The eval printed head beats zero-shot: True (+0.1123) when the matched comparison was +0.0243. The fix extracted the decode into one function both paths call (specialists/grounding/inference.py::decode_candidates_from_features), and re-running the eval reproduced the Phase 7 number exactly (|difference| = 0.0000) — proof the fix is real, not cosmetic.

Status: MEASURED. The trained head beats the zero-shot baseline; the absolute IoU is low, which is an honest limitation. See EVALUATION.md §2.2 and MODELS.md §3.4.

---

4.3 Optical-SAR fusion — MEASURED, ruling OPEN

• Artifact: artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json.
• Tool: scripts/eval_fusion_115.py (read-only) — a separate, later step from training.
• Split: held-out test, n = 4,000 (n_scored: 4000, split: "test").
• Label space: 19 CLC classes; 14 present, 5 absent in the scored split.
• macro-F1 denominator: all 19 classes (absent classes contribute 0.0) — recorded explicitly.

Why the metric is computed by a separate tool. Every run_record.json carries pre_registered_metric_computed = false and result_status = "PLUMBING_ONLY — fixture/loop evidence, NOT a result; the pre-registered 11.5 metric is not computed". That flag is correct about the trainer and must not be "fixed": the trainer fits on train/val and never opens the test split, deliberately, so the held-out split cannot be contaminated by the search over 10 runs, 2 arms and 5 seeds. The 11.5 metric therefore belongs to a separate, read-only step.

The tool refuses (exit 2) rather than guessing when: the head or cache is missing; the split is empty; the cache config_hash is not 78f1e3700da15aa1 (a different experiment); or the head cannot be built against the frozen (B, 2318) input contract.

Measured numbers:

Metric	Value	Exact key path
accuracy	0.931	accuracy
macro F1	0.434161	macro_f1
loss	0.254592	loss
n scored	4,000	n_scored
num classes	19	num_classes
macro-F1 denominator	all 19 classes (absent contribute 0.0)	macro_f1_denominator
classes present	[0,2,3,4,5,6,7,8,9,10,12,13,17,18]	classes_present
classes absent	[1,11,14,15,16]	classes_absent
deciding statistic	false	is_deciding_statistic
head sha256	785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab	head_sha256
head bytes	14,427,457	head_bytes
cache arm	A	cache_arm
cache config hash	78f1e3700da15aa1	cache_config_hash

Optical-SAR accuracy 0.931 must ALWAYS travel with macro-F1 0.434161. The two are recorded side by side in the artifact precisely so neither can be quoted alone.

Per-class F1 (_per_class_f1, 19 terms):

[1.000, 0.000, 0.428, 0.571, 0.887, 0.000, 0.000, 0.800, 0.929, 0.940,
 0.182, 0.000, 0.200, 0.438, 0.000, 0.000, 0.000, 0.877, 0.998]

Two averages from the same vector — the denominator trap:

Averaged over	Value
all 19 slots — the pre-registered definition	0.434161
the 14 present classes only	0.589218

A reader who computes the second and compares it to the recorded scalar will wrongly conclude the recorded figure is wrong. It is the 19-slot mean by construction. Reproducible: sum(per_class)/19 = 0.43416082670034284, sum(per_class[c] for c in present)/14 = 0.5892182648076082 (docs/PHASE12_115_METRIC_COMPUTED.md §3.5).

Five absent classes contribute 0.0 by definition — but two PRESENT classes also score 0.0. Classes 1, 11, 14, 15, 16 are absent; classes 5 and 6 are present and score 0.0 — a real total miss. So part of the low macro-F1 is populational, and part is genuine per-class failure. High accuracy with a wide per-class spread is the signature of prediction dominated by frequent classes.

Measured test-set label distribution (docs/PHASE12_115_METRIC_COMPUTED.md §3.4):

[13, 0, 63, 39, 846, 4, 1, 6, 453, 246, 4, 0, 9, 18, 0, 0, 0, 34, 2264]

14 of 19 classes present; five have zero samples; top-to-bottom ratio 2,264 : 1 (class 18 vs class 6). The majority class holds 2,264 / 4,000 = 0.566; the head scores 0.931, so the accuracy is not a constant predictor. Per-class recall (selected): class 18 n=2264 → 0.997; class 9 n=246 → 0.951; class 17 n=34 → 0.941; class 8 n=453 → 0.905; class 4 n=846 → 0.809; class 3 n=39 → 0.410; class 12 n=9 → 0.111; classes 5 and 6 → 0.000.

Four independent verification checks (docs/PHASE12_115_METRIC_COMPUTED.md §3): (1) evaluate_fusion_head directly returns n=4000, accuracy=0.931, macro_f1=0.43416082670034284; (2) rebuilding the input tensor by hand — concat([optical_gap, sar_gap, joint_gap]) then concat([…, optical_mask, sar_mask]), widths (4000,768)×3 + (4000,12) + (4000,2) = 2318 — gives 0.931 again (this checks feature ordering); (3) sklearn.metrics accuracy_score = 0.931, f1_score(average="macro", labels=range(19), zero_division=0) = 0.43416082670034284; (4) the per-class breakdown above.

⚠️ The caveat that governs how these numbers may be used. The cache metadata records label_policy = require_single_label, n_skipped_by_policy = 0. reBEN v2.0 is a multi-label corpus while the frozen head is a single-label 19-class softmax. The extraction used policy (b) — restrict to single-label patches — which preserves the frozen architecture exactly but changes the evaluation population. Single-label patches are 17.57 % of the corpus (96,537 of 549,488), and the rarest class survives as 1 patch (a 59,204 : 1 imbalance). So this metric may not be presented as a multi-label BigEarthNet/reBEN result, as comparable to published BigEarthNet numbers, or as a statement about all 19 classes. It may be presented as the pre-registered 11.5 metric under the single-label extraction policy on the held-out split.

Status: MEASURED, ruling OPEN. is_deciding_statistic: false; the artifact's advisory states it "selects no head, ranks nothing and compares no arms. Whether this constitutes a Phase 12 pass is the owner's ruling." No numerical threshold is attached. See EVALUATION.md §2.3 and MODELS.md §3.3.

---

4.4 Change-VQA — MEASURED, ruling OPEN

• Artifact: artifacts/change_vqa/run/PROMOTION.json (verification block).
• Splits: test (n = 39,686) and test2 (n = 31,036).
• Selection: epoch 8, chosen on val answer accuracy 0.700018, stopped by early stopping (identity.selected_on, identity.stop_reason).
• Answer space: a closed 19-answer space over the eight CDVQA question types; type mask applied by default (verification.mask_gain: 0.0).

Two test sets are reported. Quoting only the better one would be selective.

Test set	n	accuracy	macro F1
test	39,686	0.697626	0.378373
test2	31,036	0.651469	0.372309

Field	Value	Exact key path
test accuracy	0.697626367	verification.test_accuracy
test macro F1	0.378373275	verification.test_macro_f1
test2 accuracy	0.651469262	verification.test2_accuracy
test2 macro F1	0.372308516	verification.test2_macro_f1
n scored, test	39,686	verification.n_scored_test
n scored, test2	31,036	verification.n_scored_test2
global majority baseline, test	0.311546	verification.global_majority_baseline_test
global majority baseline, test2	0.178728	verification.global_majority_baseline_test2
mask gain	0.0	verification.mask_gain
checks passed / failed / unverified	93 / 0 / 0	verification.checks_*
metric ruling	OPEN — the plan leaves the accuracy/macro-F1 interpretation owner-gated. No official aggregate metric is asserted here.	verification.metric_ruling

The gap between accuracy and macro-F1 means the head is carried by common classes. The global-majority baselines make it concrete: 0.311546 on test means a constant predictor scores 0.31 and the head scores 0.70 — but the macro-F1 of 0.378 shows the per-class picture is far weaker than the aggregate.

Status: MEASURED, ruling OPEN. No promotion/acceptance decision has been recorded (PROMOTION.json → state.note). See EVALUATION.md §2.4 and MODELS.md §3.2.

---

4.5 VLM adapter — MEASURED, ACCEPTANCE-REJECTED

• Artifact: artifacts/vlm/phase6_closure.json (status: CLOSED).
• Split: a frozen 1,000-question subset of the test split, whose identity was proven without a model (Gate A″): available_per_split {val: 6750, test: 7772}, subset n = 1000, test per-class counts an exact match to the recovery subset (19 classes, sum 1000).

The "usable" numbers (why_usable_verified.adapted_test):

Metric	Value	Exact key path
exact_match	0.963	why_usable_verified.adapted_test.exact_match
F1	0.96432	why_usable_verified.adapted_test.f1
n	1000	…adapted_test.n
precision	0.963391	…adapted_test.precision
recall	0.965251	…adapted_test.recall
tp / fp / tn / fn	500 / 19 / 463 / 18	…adapted_test.confusion
aggregate test delta	+49.50 pp (46.80 → 96.30 pp)	why_usable_verified.aggregate_test_delta_pp

The "rejected" side (why_acceptance_rejected, rule version v002, decision split test):

V1	passes — test 46.80 → 96.30 pp, delta +49.50 pp (required ≥ +5.00)
V2	fails — Mixed forest: n=33, 100.00 → 87.8788 pp, drop 12.1212 pp, lost_questions 4, z 2.1335

It fails both halves of v002 (lost ≥ 4 and z ≥ 1.96). Per item V, a complete run that fails V2 is REJECTED. The rejection is narrow: n_classes_failed: 1, n_classes_held: 5, n_classes_improved: 11, n_classes_total: 19. The next-worst class (Inland wetlands, −6.6667 pp, n=30) lost only 2 questions and sits below V2's materiality floor.

It is not a split artefact. The same class also degraded on val in Run 1 (drop 6.4516 pp, n=31). The adapter hurts Mixed forest on both splits. Mixed forest also sits at a 100.00 pp baseline on test, so any loss is a drop from the ceiling.

Residual risk, reported not resolved. The verdict rests on 4 questions in one class of 33. With no n ≥ N floor in V2, a 33-question class can flip the verdict of a run whose aggregate endpoint improved by 49.5 pp.

Both rejection records are preserved (preserved_records): Run 1's manifest v001 on val REJECTED (3 classes, val_delta_pp 42.0, test_delta_pp null); the independent test-split adjudication v002 on test REJECTED; and the recovery manifest v002 on val ACCEPTED — recorded for completeness only, NOT final acceptance, because it decides on the same val subset that motivated v002.

Status: MEASURED; artifact CLOSED; headline ACCEPTANCE-REJECTED. USABLE ≠ ACCEPTED. The deployed caption/VQA path uses the unadapted model. See EVALUATION.md §2.5 and MODELS.md §3.6.

---

4.6 Router — MEASURED, TEST NOT RUN

• Artifact: artifacts/router/threshold_sweep_val.json.
• Split scored: validation, n = 86 (n_val: 86, n_val_examples_scored: 86, n_test_examples_scored: 0, test_split_touched: false).
• Metric: overall ungated accuracy 0.965116 (overall_ungated_accuracy).
• Corpus: corpus-limited (corpus_limited: true); corpus_total: 576, corpus_groups: 54; plan minima plan_min_val_queries: 500, plan_min_hard_negatives: 100; val_min_support: 8; hard_negatives_in_val: 0 (by design — hn_* families are held out to TEST).
• Sweep: thresholds 0.50 → 0.99; select_by: "covered_accuracy"; selected.threshold: 0.76 (coverage 0.790698, covered-task accuracy 1.0); shipped_threshold: 0.70 (coverage 0.848837, covered-task accuracy 0.972603); delta_vs_shipped: {coverage: -0.0581, covered_task_accuracy: 0.0274}.
• Split sizes: {train: 410, test: 80, val: 86}.

The artifact's own note, quoted verbatim in substance: "corpus-limited: val n=86 vs plan >=500. This is NOT a calibration — the corpus is synthetic and too small (min per-class support 8, caption) and val carries 0 hard negatives (hn_ families are held out to TEST by design). Selecting a threshold here yields a justified default, not a calibrated value. The corpus was NOT padded with generated queries. Backlog P1-9's 'n=80' is the TEST split; the sweep target is val n=86. The test split was NOT touched."*

The router TEST split was NOT RUN. Do not read 0.965116 as a test result. The sweep scored 86 validation examples and zero test examples.

Historical record, kept distinct. docs/PHASE4_ROUTER_REPORT.md (verified 2026-09-16, Gate 2 PASS) records an earlier gate-2 evaluation that included a test column — test task accuracy 0.975, macro F1 0.976, hard-negative accuracy 0.800. That evaluation predates the shipped threshold sweep, and its own standing caveat is that "the 0.975 headline is partly earned on templates the split kept in training. Treat the router as working, not as benchmarked." The release's position for the shipped artifact is TEST NOT RUN; the Phase 4 numbers are retained as a historical record, not promoted to a release benchmark.

Status: MEASURED — TEST NOT RUN. See EVALUATION.md §2.6 and MODELS.md §3.5.

---

4.7 Calibration — MEASURED, negative result

• Artifact: artifacts/calibration_v001.json (schema: calibration_v1).
• Fit split: val, n = 16,441 (temperature_scaling.fitted_on: "Val", temperature_scaling.n_samples: 16441).
• Temperature: T = 0.9772731820958189 (temperature_scaling.temperature).
• Scope: the R-02 change-VQA head's answer confidence only; other specialists emit their own raw scores and are unaffected (scope.note).

Metric	Before	After	Improvement	Key path
ECE	0.013755	0.014929	−0.001174	metrics.ece_before / metrics.ece_after / metrics.ece_improvement
NLL	0.689741	0.689631	+0.00011	metrics.nll_before / metrics.nll_after

ece_improvement = −0.001174 is negative: expected calibration error increased. NLL improved by a negligible 0.00011 (fit_diagnostics.nll_improvement: 0.0001103574982127542). metrics.n_bins: 15, metrics.n_classes: 19.

The scaling is retained because it is part of the frozen configuration, not because it helped. provenance.held_out_splits_excluded: ["Test", "Test2"] — the temperature was fitted only on val.

The reliability diagram is explicitly labelled pre-scaling. reliability_diagram.ece is 0.013755 (the before value), with the note: "Equal-width bins over predicted-class confidence. ECE is bin-count sensitive and is not an aggregate score." The diagram on the Benchmark page is labelled pre-scaling so a reader cannot mistake it for the calibrated result.

Status: MEASURED — worse. A negative result, kept negative. See EVALUATION.md §2.7 and MODELS.md §5.

---

5. Live validation — behavioural, not accuracy

Accuracy is separate from behavioural validation. The deployed stack was driven end-to-end in a headed browser, one upload per case, with per-case screenshots and recorded run ids.

Property	Result
Independent full passes	3
Cases per pass	8 (6 regression + 2 router-defect)
Passes at 8/8	3 of 3
Live runs executed	24
Correct dispatches	24
Mock-node contamination	0 on every live run
Trace fill	94.4444 % on every live run
Frontend regression suite	106 passed (tests/unit/test_frontend_live_wiring.py)

3 passes × 8 cases = 24 runs, 8/8 each, 0 mock nodes, 94.4444 % trace fill. Every pass produced fresh run identifiers — no run id is shared between passes. This is behavioural evidence that the pipeline runs and routes correctly; it is not an accuracy claim.

5.1 Harness integrity — a false-pass bug that was caught

An earlier harness revision typed queries with synthetic CDP key events, which Chrome silently drops when the window lacks OS focus. The harness therefore dispatched the page's default query and still recorded a "result" — a false pass.

The current harness asserts form state before dispatch: that the query box really holds the intended query, that #obsTail reads ready, and that both frames are attached for pair tasks. The earlier 8/8 run was independently checked and confirmed not to have been infected (its answers were query-specific and the query text was embedded in the answers). This failure mode is recorded because it is exactly the kind of silent false-positive an evaluation harness must not have.

5.2 Deployed HEADs

The live validation ran against these deployed revisions: Frontend 2d7ae53b482d, Backend 89d80eaddec5, Inference 5a0936ace491.

5.3 What live validation does and does not establish

• Establishes: the pipeline runs end-to-end; the router dispatches the right specialist for the 8 cases in each pass; no mock nodes contaminate a live run; the trace is populated to 94.4444 %.
• Does not establish: any accuracy figure. There is no labelled ground truth attached to the 24 live runs, so no accuracy can be computed from them.

---

6. What is NOT benchmarked

An exhaustive list. Absent numbers are stated, never silently omitted or replaced with an estimate.

Benchmark	Status	Note
System-level end-to-end accuracy	NOT RUN — none exists	There is no measured end-to-end benchmark of the full router → specialist → envelope pipeline. No such number is claimed anywhere.
Router test split	NOT RUN	Only the validation split (n = 86) was scored by the shipped threshold sweep.
Grounding at 448 for the trained head	NOT RUN	Only the zero-shot baseline was measured at 448 (docs/MODELS.md §4.1). Re-opening the resolution question for the trained head would be a new pre-registered experiment.
Benchmark adapters	NOT RUN	Adapter-based benchmark runs were not executed.
Efficiency / latency benchmark	NOT RUN (system-level)	Per-specialist latency is recorded incidentally in artifacts (e.g. grounding latency_ms_per_image 2.205 ms for the threshold decode, 0.655 ms for argmax; CROMA forward pass 0.89 s for a batch of 2 at 120 px), but there is no end-to-end latency benchmark.
Cross-dataset generalisation	NOT RUN	Each specialist is evaluated only on its own training-family test split.
Human evaluation	NOT RUN	No human rating of answers was performed.
Robustness / adversarial evaluation	NOT RUN	No perturbation, adversarial, or out-of-distribution robustness study was performed.
Multi-label BigEarthNet / reBEN result	NOT PRODUCED	The local subset is 100 % single-label against the official 1–11 multi-label scheme. Metrics are not comparable to published multi-label numbers.
Grounding head wired and benchmarked in the serving path	NOT RUN	specialists/grounding/inference.py exposes only the zero-shot path as of the Phase 8 record; wiring the head is the remaining integration step.
Change-VQA served end-to-end	NOT RUN	Serving needs both a trained head and a trained feature extractor; neither is wired by default, and an untrained head refuses to answer.
Optical-SAR multi-label evaluation	NOT RUN	See the single-label caveat (§4.3).
Change-VQA on a third held-out set	NOT RUN	Only test and test2 exist.
Any private / hidden evaluation data	NOT ACCESSED	evaluation.hidden_data_access: false in configs/base.yaml.
Calibrated reliability curve	NOT PLOTTED	Only the pre-scaling diagram exists; the calibrated curve is not plotted.
Per-class support for the router test split	NOT RUN	The test split was not scored, so it has no support table.

---

7. How to reproduce

7.1 Verify every number in this document against its artifact

# verify every README/benchmark number against its source artifact
python release/tools/verify_readme_metrics.py

The script resolves nested artifact keys — including keys that themselves contain dots, such as the recall dict keyed "0.10"/"0.25"/"0.50" — and compares each value at the precision printed in the README. It exits non-zero if any claim fails, and prints ALL CLAIMS VERIFIED (exit 0) only when all 20 numeric claims match and the status assertions hold. Committed output: ../tools/readme_metrics_report.txt.

7.2 Reproduce the individual metrics

Metric	Reproduction path
change	scripts/eval_change.py (refuses to score on a config-hash drift, exit 3)
grounding	scripts/exp_grounding_resolution.py + scripts/analyze_grounding_resolution.py; the two eval JSONs are recomputable from the per-sample JSONL without re-running the encoder
optical-SAR	python scripts/eval_fusion_115.py --head … --cache-dir artifacts/optical_sar/fusion_features --split test --report … (exit 0 = computed; exit 2 = refused)
change-VQA	scripts/evaluate_change_vqa.py (default checkpoint artifacts/change_vqa/run/head.pt)
VLM	scripts/phase6_close.py regenerates the closure record from the evidence; scripts/phase6_adjudicate_test.py re-runs the independent test-split adjudication
calibration	load_calibration(config, base_dir='configs') consumes calibration_v001.json
router	scripts/train_router.py (cached-embedding training); the sweep artifact is threshold_sweep_val.json

See REPRODUCIBILITY.md for the full reproduction guide and EVALUATION.md for how each number was produced.

7.3 Reproduce the live validation

The 24 live runs were driven in a headed browser with per-case screenshots and recorded run ids. The harness asserts form state before dispatch (§5.1). See REPRODUCIBILITY.md for the harness entry point.

7.4 Verify the released artifacts

sha256sum -c models/checksums.sha256

---

8. The verification tool — exactly what it checks

release/tools/verify_readme_metrics.py is read-only: it writes nothing and prints a report suitable for pasting into a release-verification record. It holds a hard-coded CLAIMS list of (label, artifact path, dotted key, printed value) tuples. Key names were discovered by walking the artifacts, not assumed — several live under nested paths (change metrics are metrics.pooled.iou; grounding is results.head_threshold.mean_best_iou; VLM is why_usable_verified.adapted_test.*).

8.1 The 20 numeric claims

#	Claim	Artifact	Key path	Printed
1	change pooled IoU	artifacts/change/eval_test/eval_result.json	metrics.pooled.iou	0.8122
2	change macro IoU	…/eval_result.json	metrics.macro.miou	0.8457
3	change pooled F1	…/eval_result.json	metrics.pooled.f1	0.8964
4	grounding canonical head_threshold mean_best_IoU	…/eval_result_canonical.json	results.head_threshold.mean_best_iou	0.2838
5	grounding canonical head_threshold recall@0.5	…/eval_result_canonical.json	results.head_threshold.recall.0.50	0.2198
6	grounding matched6 head_threshold mean_best_IoU	…/eval_result_matched6.json	results.head_threshold.mean_best_iou	0.2566
7	grounding matched6 head_threshold recall@0.5	…/eval_result_matched6.json	results.head_threshold.recall.0.50	0.1938
8	grounding head_argmax mean_best_IoU (canonical)	…/eval_result_canonical.json	results.head_argmax.mean_best_iou	0.1215
9	grounding zero-shot baseline IoU (canonical)	…/eval_result_canonical.json	results.zero_shot_matched.mean_best_iou	0.0972
10	optical-SAR fusion accuracy	…/pre_registered_115_metric.json	accuracy	0.931
11	optical-SAR fusion macro_F1	…/pre_registered_115_metric.json	macro_f1	0.434161
12	change_vqa test accuracy	artifacts/change_vqa/run/PROMOTION.json	verification.test_accuracy	0.697626
13	change_vqa test macro_F1	…/PROMOTION.json	verification.test_macro_f1	0.378373
14	change_vqa test2 accuracy	…/PROMOTION.json	verification.test2_accuracy	0.651469
15	change_vqa test2 macro_F1	…/PROMOTION.json	verification.test2_macro_f1	0.372309
16	router overall ungated accuracy	artifacts/router/threshold_sweep_val.json	overall_ungated_accuracy	0.965116
17	calibration ECE before scaling	artifacts/calibration_v001.json	metrics.ece_before	0.013755
18	calibration ECE after scaling	artifacts/calibration_v001.json	metrics.ece_after	0.014929
19	VLM adapter exact_match	artifacts/vlm/phase6_closure.json	why_usable_verified.adapted_test.exact_match	0.963
20	VLM adapter F1	artifacts/vlm/phase6_closure.json	why_usable_verified.adapted_test.f1	0.96432

8.2 The status assertions

Beyond the 20 numeric claims, the tool asserts artifact-declared statuses:

VLM headline contains ACCEPTANCE-REJECTED : True
VLM status                               : CLOSED
router corpus_limited                    : True
router n_val                             : 86
calibration temperature                  : 0.9772731820958189
calibration ece_improvement              : -0.001174  (negative => calibration did NOT help)

8.3 The committed output

release/tools/readme_metrics_report.txt records all 20 rows as MATCH, all six status assertions as expected, and the final line:

RESULT: ALL CLAIMS VERIFIED

The tool exits 0 on that result and 1 otherwise, so a drifted artifact fails the release check rather than passing silently.

8.4 What the tool does not check

• It does not check the n or split fields. A value could match while the population changed; the n/split columns in §1 are therefore maintained by hand against the artifacts.
• It does not check the calibration reliability diagram, the grounding per-sample JSONL, or the VLM per-class tables.
• It does not verify the live-validation record (§5), which is behavioural and not artifact-scored.

---

9. Artifact schemas and key-path resolution

Every metric in §1 is addressed by a dotted key path into a JSON artifact. Two mechanics matter, because getting either wrong produces a NOKEY failure that looks like a missing metric rather than a bug in the reader.

9.1 The dotted-key problem

Some artifact keys themselves contain dots. The grounding artifacts key their recall dict as {"0.10": …, "0.25": …, "0.50": …}. A naive dotted.split(".") walk would break results.head_threshold.recall.0.50 into … recall → 0 → 50 and fail — even though the value exists.

verify_readme_metrics.py resolves this by preferring the longest matching key at each step: at a dict node it tries the longest possible joined key first, and only shortens when no key matches. That is why the recall claim is written results.head_threshold.recall.0.50 and resolves to the "0.50" entry rather than to a nested 0 object.

9.2 Top-level schema of each artifact family

These are the top-level keys actually present in each artifact, read from the files:

Artifact	Top-level keys (abridged to the load-bearing ones)
change/eval_test/eval_result.json	artifact, checkpoint, checkpoint_config_hash, checkpoint_config_hash_checked, checkpoint_embedded_config, config_drift, config_hash, created_at, device, environment, metrics, n, n_images_with_change, change_fraction_quantiles, per_image_change_fraction
grounding/…/eval_result_canonical.json	checkpoint_config_hash, config_drift, config_hash, created_at, device, environment, eval_cache, frozen_config_evaluation, grid, head_decode, limited_run, n_eval_records, phase7_reference, resolution, results
grounding/…/eval_result_matched6.json	same shape; head_decode.top_k differs (6 vs 20)
optical_sar/…/pre_registered_115_metric.json	tool, metric, definition, split, head_path, head_sha256, head_bytes, cache_path, cache_arm, cache_config_hash, n_scored, accuracy, macro_f1, loss, num_classes, macro_f1_denominator, classes_present, classes_absent, _per_class_f1, is_deciding_statistic, advisory
change_vqa/run/PROMOTION.json	schema, promoted_utc, promoted_by, artifact, source, identity, frozen_dependency, verification, state, serving_wiring
router/threshold_sweep_val.json	adapter_config_hash, adapter_encoder, adapter_path, artifact, config_hash, corpus_groups, corpus_limited, corpus_total, created_at, delta_vs_shipped, device, encoder_type, environment, force, hard_negatives_in_val, n_test_examples_scored, n_val, n_val_examples_scored, note, overall_ungated_accuracy, plan_min_hard_negatives, plan_min_val_queries, rows, seconds, select_by, selected, shipped_row, shipped_threshold, split, split_sizes, test_split_touched, thresholds, val_min_support, val_task_counts
calibration_v001.json	consumer_contract, created_utc, fit_diagnostics, metrics, provenance, reliability_diagram, schema, scope, temperature_scaling, type_mask_applied
vlm/phase6_closure.json	artifact_verification, evidence_index, forward_rule, generated_at, headline, kind, not_a_retrain, phase, preserved_records, production_adapter, rule_unchanged_by_closure, status, verified_vs_accepted, what_closure_does_not_claim, why_acceptance_rejected, why_usable_verified

9.3 Status and provenance fields carried by every artifact

The artifacts are not just numbers; each carries the fields that make its number auditable:

Field	Meaning	Where
config_hash	the frozen registry hash the run was made against (78f1e3700da15aa1)	change, grounding, optical-SAR, change-VQA, router, calibration, VLM
checkpoint_config_hash	the config hash baked into the weights	change, grounding
config_drift	whether the two hashes disagreed at evaluation time	change, grounding (both false)
created_at / generated_at	when the artifact was produced	all
device / environment	device and library versions	change, grounding, router
split	the population scored	optical-SAR (test), router (val)
n / n_scored / n_eval_records	the population size	all

Why config_drift: false matters. The evaluation is only valid against the configuration the weights were trained under. checkpoint_config_hash_checked: true and config_drift: false together record that the check was performed and passed — not merely omitted.

9.4 The is_deciding_statistic and ruling fields

Two artifacts carry an explicit statement that their number is not a decision:

• optical-SAR: is_deciding_statistic: false, with an advisory that the tool "selects no head, ranks nothing and compares no arms."
• change-VQA: verification.metric_ruling: "OPEN — … No official aggregate metric is asserted here."

These fields exist so that a later reader cannot promote a measurement into a ruling by quoting it out of context. Where a ruling exists, it is recorded separately (e.g. the optical-SAR A/B arm ruling was made on best_val_accuracy — A 0.837100 vs B 0.839100, floor 0.0285 → Arm A retained, ruling R-14 — not on the 0.931/0.434161 held-out metric).

---

10. Metric glossary and aggregation arithmetic

Definitions, so that a reader can recompute every number from the stored confusion counts or per-class vectors.

10.1 Pooled vs macro

Term	Definition	Where used
pooled IoU	IoU over the union of all pixels, computed from the global tp/fp/fn counts	change (metrics.pooled.iou = 0.8122)
macro IoU	unweighted mean of per-class IoU. For binary change this is the mean over {change, no-change}	change (metrics.macro.miou = 0.8457)
pooled F1	2·tp / (2·tp + fp + fn) from the global counts	change (metrics.pooled.f1 = 0.8964)
macro F1	unweighted mean of per-class F1	change (metrics.macro.f1 = 0.7962); optical-SAR (macro_f1 = 0.434161)

For the change artifact, metrics.macro.iou = 0.718 is the change-class IoU, while metrics.macro.miou = 0.8457 is the mean over both classes — these are different numbers from the same block, and the headline uses miou.

10.2 Mean best IoU, mean matched IoU, recall@IoU

Term	Definition	Where used
mean best IoU	for each record, the maximum IoU over all predicted boxes; then the mean over records	grounding (mean_best_iou = 0.2838 / 0.2566 / 0.1215 / 0.0972)
mean matched IoU	the IoU of the matched prediction only, so unmatched records contribute 0	grounding (mean_matched_iou, equal to mean_best_iou in all three strategies)
recall@0.10 / 0.25 / 0.50	fraction of records whose best predicted box reaches that IoU	grounding (results.*.recall)

Mean best IoU is a max over predictions, so emitting more boxes raises it mechanically. That is exactly why the candidate count is recorded (head_decode.top_k) and why the decode-matched protocol (top_k 6 vs the baseline's 5.99) is reported alongside the config default (top_k 20). See §4.2.

10.3 The macro-F1 denominator

For a K-class head, macro_f1 = (1/K) · Σ_c F1_c. The denominator is the label space, not the number of classes present in the scored split.

• optical-SAR: K = 19 (num_classes), 14 present, 5 absent. Absent classes contribute 0.0. All-19 mean = 0.434161; present-14 mean = 0.589218.
• The artifact records macro_f1_denominator: "all 19 classes (absent classes contribute 0.0)", classes_present, classes_absent, and the full 19-term _per_class_f1 vector so the denominator is auditable rather than assumed.

10.4 Percentage points vs fractions

Some artifacts record a fraction (0.963) and some a percentage-point delta (49.5 pp). The VLM record uses both: exact_match: 0.963 is a fraction, while aggregate_test_delta_pp: 49.5 and the baseline/adapted 46.80 → 96.30 pp are percentage points. A reader must not confuse +49.5 pp with +0.495 of a fraction — the former is a 49.5-point move on a 0–100 scale.

10.5 ECE and NLL

Term	Definition	Where used
ECE	expected calibration error: bin-count-sensitive average gap between confidence and accuracy over equal-width bins	calibration (metrics.ece_before = 0.013755, metrics.ece_after = 0.014929, n_bins = 15)
NLL	mean negative log-likelihood	calibration (metrics.nll_before = 0.689741, metrics.nll_after = 0.689631)

ECE is bin-count sensitive and is not an aggregate score — recorded verbatim in the artifact's reliability_diagram.note. A smaller ECE is better; here it increased (ece_improvement = −0.001174), which is a negative result.

10.6 Coverage, fallback rate and ungated accuracy (router)

Term	Definition	Where used
overall ungated accuracy	task accuracy over all examples, whether or not the confidence gate fired	router (overall_ungated_accuracy = 0.965116)
coverage	fraction of examples the gate accepted	router (rows[*].coverage; shipped 0.848837 at threshold 0.70)
covered task accuracy	task accuracy among gated-in examples only	router (rows[*].covered_task_accuracy; shipped 0.972603 at 0.70)
fallback rate	fraction routed to the deterministic lexical fallback	router (rows[*].fallback_rate; shipped 0.151163 at 0.70)

The headline is the ungated figure, because a gated accuracy without its coverage is not comparable across thresholds. That is why the sweep reports both, and why "ungated" is part of the label.

10.7 Why the router's selected threshold differs from the shipped threshold

select_by: "covered_accuracy" maximises accuracy among gated-in examples, which is monotone in the threshold (at 0.76 the covered accuracy is 1.0 on 68 examples). The shipped threshold is 0.70 (coverage 0.848837, covered accuracy 0.972603), which is the config default. The delta_vs_shipped block records the difference: {coverage: -0.0581, covered_task_accuracy: 0.0274} — i.e. the selected threshold covers 5.81 points fewer examples for 2.74 points more covered accuracy. The artifact is explicit that this is "a justified default, not a calibrated value."

---

11. Evidence index

Topic	Evidence
Verification tool	release/tools/verify_readme_metrics.py; committed output release/tools/readme_metrics_report.txt
Change	artifacts/change/eval_test/eval_result.json
Grounding (canonical / matched6)	artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json, …/eval_result_matched6.json; docs/PHASE7_RESOLUTION_DECISION.md, docs/PHASE8_GROUNDING_HEAD_DECISION.md
Optical-SAR	artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json; docs/PHASE12_115_METRIC_COMPUTED.md, docs/PHASE14_OPTICAL_SAR_DECISIONS.md
Change-VQA	artifacts/change_vqa/run/PROMOTION.json
Router	artifacts/router/threshold_sweep_val.json; docs/PHASE4_ROUTER_REPORT.md
Calibration	artifacts/calibration_v001.json
VLM	artifacts/vlm/phase6_closure.json; docs/PHASE6_CLOSURE.md
Live validation	the live-validation record and per-case screenshots; tests/unit/test_frontend_live_wiring.py (106 passed)
Cross-links	EVALUATION.md (how each number was produced), MODELS.md (the artifacts behind each number), DATASETS.md (the corpora), TRAINING.md (how each head was trained), LIMITATIONS.md (the honest catalogue of gaps), REPRODUCIBILITY.md (reproduction guide)