release: add docs/REPRODUCIBILITY.md

docs/REPRODUCIBILITY.md

@@ -1,130 +1,1239 @@
 # Reproducibility
 
-**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `NOT RUN` · `BLOCKED`.
+**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `ATTEMPTED` · `NOT RUN` · `BLOCKED` ·
+`DEFERRED` · `REJECTED` · `OPEN` · `RESOLVED` · `CLOSED`.
 
-This document states exactly what a third party can reproduce, with what, and what they cannot.
+This document states **exactly what a third party can reproduce, with what, and — just as
+importantly — what they cannot**. It is written to the same depth as the rest of this documentation
+set: real commands, real file paths, real hashes, real expected output, and an explicit status for
+every claim.
+
+It is deliberately **not** a "quickstart". A quickstart tells you the happy path and hides the
+edges. This document records the edges, because in this project the edges are where the real
+debugging time went — a dead sandbox proxy, a pytest that only exists in one virtualenv, a Chrome
+that silently drops synthetic key events, a Hugging Face helper that returned an *empty file* and
+would have produced a **false failure**. Those are documented in §10 so that a third party does not
+rediscover them.
+
+> **The one rule this document obeys, like every document in `docs/`:** *do not fabricate*. Every
+> claim below is grounded in a file that was read (`core/config.py`, `configs/base.yaml`,
+> `release/tools/*.py`, `artifacts/**`, `docs/**`). Where the evidence does not establish something,
+> the text says `UNKNOWN — not established from the available evidence`.
+
+---
+
+## Table of contents
+
+1. [The reproducibility contract](#1-the-reproducibility-contract)
+2. [The frozen configuration](#2-the-frozen-configuration)
+3. [Reproduce the metric verification (cheap, no GPU)](#3-reproduce-the-metric-verification-cheap-no-gpu)
+4. [Reproduce the environment](#4-reproduce-the-environment)
+5. [Reproduce the tests](#5-reproduce-the-tests)
+6. [Reproduce a live run](#6-reproduce-a-live-run)
+7. [Reproduce and verify the released artifacts](#7-reproduce-and-verify-the-released-artifacts)
+8. [What "reproduce" means for the two externally-trained artifacts](#8-what-reproduce-means-for-the-two-externally-trained-artifacts)
+9. [What is NOT reproducible from this release — exhaustive](#9-what-is-not-reproducible-from-this-release--exhaustive)
+10. [Environment traps recorded for reproducibility](#10-environment-traps-recorded-for-reproducibility)
+11. [What is NOT RUN / OPEN / BLOCKED for this topic](#11-what-is-not-run--open--blocked-for-this-topic)
+12. [Where the evidence lives](#12-where-the-evidence-lives)
 
 ---
 
 ## 1. The reproducibility contract
 
-| Guarantee | How it is enforced |
-|---|---|
-| **Frozen configuration** | All tunables live in `configs/base.yaml`; no magic numbers in Python. The loader validates invariants and computes a hash. |
-| **Frozen config hash** | `78f1e3700da15aa1`. Every artifact records the hash it was produced against. |
-| **Pinned backbones** | Every backbone is pinned by revision in `configs/base.yaml`; the Hub resolves the exact commit. |
-| **Seed** | `project.seed: 42`. |
-| **Immutable public test** | `evaluation.immutable_public_test: true`; `hidden_data_access: false`. |
-| **Byte-verified artifacts** | Every released artifact ships with a sha256 in `models/checksums.sha256`. |
-| **Verified metrics** | Every quoted number is checked against its artifact by `tools/verify_readme_metrics.py`. |
+Reproducibility here is not a slogan; it is a set of **enforced guarantees**. Each row below states a
+guarantee, how it is enforced in code or in a tool, and how a third party can independently check
+it.
+
+| Guarantee | How it is enforced | How to check it |
+|---|---|---|
+| **Frozen configuration** | All tunables live in `configs/base.yaml`; there are no magic numbers in Python. The loader validates invariants and computes a hash. | Read `configs/base.yaml`; run the loader; compare the hash. |
+| **Frozen config hash** | `Config.hash` = `sha256(json.dumps(data, sort_keys=True, default=str))[:16]` (`core/config.py:76-80`). Every evaluation run and every artifact records the hash it was produced against. | `Config.hash` returns `78f1e3700da15aa1`; the same string appears in `models/manifest.json` and each artifact. |
+| **Pinned backbones** | Every backbone is pinned by `repo_id` **and** `revision` (a commit prefix, not a floating tag) in `configs/base.yaml`. | The four pins are listed in §4.3; each resolves to an exact Hub commit. |
+| **Seed** | `project.seed: 42` (`core/config.py:82-84`). | Read the key; it is in the registry. |
+| **Immutable public test** | `evaluation.immutable_public_test: true` and `evaluation.hidden_data_access: false` (`configs/base.yaml:273-277`). | Read the two keys; the corpus module refuses to mutate a sealed corpus. |
+| **Leakage isolation** | `evaluation.leakage_split_key: scene_id` — splits are assigned by *scene*, never by *tile*. | Read the key; `evaluation/leakage.py` audits manifests. |
+| **No official aggregate** | `evaluation.official_aggregate_weights: null`. There is no weighted composite score. | Read the key; `normalize.py` raises on aggregate requests. |
+| **Byte-verified artifacts** | Every released artifact ships with a sha256 in `models/checksums.sha256`, generated from the actual files. | `sha256sum -c models/checksums.sha256`. |
+| **Verified metrics** | Every number quoted in the README is checked against its source artifact by `release/tools/verify_readme_metrics.py`. | Run the tool; it prints `ALL CLAIMS VERIFIED`. |
+| **Independent Hub verification** | `release/tools/hf_verify.py` re-downloads every uploaded file over direct HTTPS and re-hashes the bytes it received. | Run the tool; it prints `artifacts failed : 0`. |
+| **Negative results preserved** | Rejected and open rulings stay in the record (`PHASE7_RESOLUTION_DECISION.md`, `phase6_closure.json`, `threshold_sweep_val.json`). | Read the artifacts; they contain the rejections, not scrubbed versions. |
+
+### 1.1 What the contract is *not*
+
+The contract does **not** promise:
+
+- a single command that retrains every artifact from scratch (§8);
+- an end-to-end system-level accuracy number (§9 — no such benchmark exists);
+- that the two externally-trained artifacts can be regenerated inside this repository (§8);
+- bit-identical *training* on arbitrary hardware (only bit-identical *released artifacts* are
+  guaranteed, by hash).
+
+### 1.2 The reproducibility ladder
+
+Reproduction here comes in four tiers, from cheapest to most expensive. A reviewer who wants to
+spend five minutes gets tier 1; a reviewer who wants to audit the whole thing walks all four.
+
+| Tier | What it proves | Cost | Section |
+|---|---|---|---|
+| **1. Metric verification** | Every quoted number matches its artifact | seconds, CPU | §3 |
+| **2. Environment + tests** | The code installs and the suites behave as recorded | minutes, CPU | §4–§5 |
+| **3. Live run** | The deployed stack answers correctly on unseen imagery | minutes (plus cold start), network | §6 |
+| **4. Artifact + Hub verification** | The released weights are byte-identical to what was uploaded | minutes, network | §7 |
+
+---
+
+## 2. The frozen configuration
+
+### 2.1 What "frozen" means here
+
+`configs/base.yaml` is the **single registry** for every tunable in the system. The design rule is:
+*no magic numbers in Python*. If a value can change behaviour — an image size, a channel count, a
+threshold, a precision, a split key — it lives in the registry, not in a function body.
+
+This matters for reproducibility because it means the **entire behavioural surface of the system is
+captured by one file**, and that file has a hash.
+
+### 2.2 The hash and how it is computed
+
+```python
+@property
+def hash(self) -> str:
+    """Stable hash of the whole registry. Recorded in every evaluation run."""
+    blob = json.dumps(self._data, sort_keys=True, default=str).encode()
+    return hashlib.sha256(blob).hexdigest()[:16]
+```
+
+— `core/config.py:76-80`.
+
+Three properties make this hash **stable**:
+
+1. **`sort_keys=True`** — key order in the YAML file does not change the hash.
+2. **`default=str`** — non-JSON-native values (dates, paths) are stringified deterministically.
+3. **`[:16]`** — the first 16 hex characters are used as the human-readable identifier.
+
+The frozen value is:
+
+```
+78f1e3700da15aa1
+```
+
+It is recorded in `models/manifest.json` (`"config_hash": "78f1e3700da15aa1"`), in each artifact's
+metadata, and in the live evidence (each live run's panel shows the hash). This is the single
+identifier that ties "the thing that produced this number" to "the configuration a reviewer is
+reading".
+
+### 2.3 The enforced invariants, with exact arithmetic
+
+Editing `configs/base.yaml` moves the hash and **invalidates every artifact keyed to it**. To make
+that failure loud rather than silent, the loader rejects a config that violates a set of recorded
+invariants. Two of them exist specifically because their violation is a *silent* error that torch
+only surfaces much later.
+
+#### Invariant 1 — fusion input dimension (finding C-1)
+
+```
+fusion.input_dim == 3 * croma.encoder_dim + croma.optical_channels + croma.sar_channels
+```
+
+With the frozen values `encoder_dim = 768`, `optical_channels = 12`, `sar_channels = 2`:
+
+```
+fusion.input_dim == 3 * 768 + 12 + 2
+                 == 2304      + 12 + 2
+                 == 2318
+```
+
+This is the concatenation of CROMA's three 768-d GAP vectors (`optical_GAP`, `SAR_GAP`,
+`joint_GAP`) plus the 12 optical channels and 2 SAR channels of the sensor adapter. The loader
+recomputes `expected` from the config and raises `ConfigError` if `fusion.input_dim` disagrees
+(`core/config.py:154-163`). The docstring is explicit about *why*: "CROMA emits optical/SAR/joint
+GAP vectors; the [declared value must match]". The upstream contract was independently reproduced
+in `docs/PHASE14_OPTICAL_SAR_DECISIONS.md §1`, where the fusion input is recomputed at runtime
+(`fusion_head.py:74`, `:311`) rather than hardcoded.
 
-### 1.1 Invariants the loader enforces
+Two related guards fire alongside it (`core/config.py:165-168`):
 
-Editing `configs/base.yaml` moves the hash and **invalidates every artifact keyed to it**. Two
-invariants exist because their violation is a *silent* error:
+```
+croma.optical_channels == 12   # CROMA s2_channels is fixed
+croma.sar_channels     == 2    # CROMA s1_channels is fixed
+```
+
+#### Invariant 2 — grounding head feature dimension (finding P7-1)
 
 ```
-fusion.input_dim            == 3 * croma.encoder_dim + croma.optical_channels + croma.sar_channels   # 2318
-grounding_head.feature_dim  == 4 * grounding.encoder_projected_dim                                  # 2048
+grounding_head.feature_dim == 4 * grounding.encoder_projected_dim
+```
+
+With `encoder_projected_dim = 512`:
+
+```
+grounding_head.feature_dim == 4 * 512 == 2048
+```
+
+The loader's own comment explains why this is **load-bearing**:
+
+> A mismatch here is a SILENT shape error. torch only raises at the [similarity step], after patch
+> features are already cached. It is therefore rejected at load time.
+
+— `core/config.py:176-195`.
+
+The "4×" arises because the grounding head consumes four projected feature streams; the projected
+dimension is declared in config precisely so this guard can exist without hardcoding 512.
+
+#### Further load-time guards
+
+The `_validate()` method (`core/config.py:94`) also enforces:
+
+| Guard | Rule | Finding |
+|---|---|---|
+| CROMA resolution | `croma.image_resolution` is an int and a multiple of 8 (CROMA asserts `% 8 == 0`) | C-7 |
+| Training precision | `training.precision ∈ {fp16, bf16, fp32}` | C-6 |
+| ZeroGPU compile ban | `deployment.torch_compile` must not be `True` — ZeroGPU does not support `torch.compile` | C-8 |
+| VLM prompt template | `vlm.prompt_must_use_chat_template` must be `true` — SmolVLM raises otherwise | — |
+
+These are the invariants a reviewer can verify by reading one method. They are the reason a
+*plausible-looking* config edit fails at load instead of producing a wrong number three hours later.
+
+### 2.4 The evaluation invariants
+
+```yaml
+evaluation:
+  immutable_public_test: true
+  hidden_data_access: false
+  official_aggregate_weights: null
+  leakage_split_key: scene_id
 ```
 
-The second is especially load-bearing: a mismatch is a silent shape error that torch only raises at
-the similarity step, after patch features are already cached. It is therefore rejected at load time.
+— `configs/base.yaml:273-277`.
+
+| Key | Value | Meaning |
+|---|---|---|
+| `immutable_public_test` | `true` | The public-test corpus is sealed; it cannot be mutated by an evaluation. |
+| `hidden_data_access` | `false` | No code path may read hidden data during a scored run. |
+| `official_aggregate_weights` | `null` | **There is no composite score.** A single "overall accuracy" is forbidden by construction. |
+| `leakage_split_key` | `scene_id` | Splits are assigned by scene, so tiles from one scene never straddle train/test. |
+
+The evaluation-honesty rules that these keys serve are stated in full in
+[`EVALUATION.md`](EVALUATION.md) §2.
 
-## 2. Reproduce the metric verification (cheap, no GPU)
+### 2.5 The deployment section is frozen paperwork
+
+```yaml
+deployment:
+  platform: huggingface-spaces
+  sdk: gradio
+  zerogpu: true
+  torch_compile: false
+  ...
+```
+
+— `configs/base.yaml:282-295`.
+
+The deployment target described here is the **superseded** HF-Space/ZeroGPU design. The active
+topology is Cloudflare Pages → Render → Codespace (`docs/DEPLOYMENT.md §1`). This section is
+retained as **frozen paperwork**: editing it would move the config hash, and the hash is what every
+artifact is keyed to. No Gradio runtime exists in code. The `torch_compile: false` line is the
+C-8 guard and remains meaningful regardless of host.
+
+### 2.6 The hash-exempt path
+
+One decision (DEV-2, the CROMA input-normalisation question) needed to become inspectable **without**
+adding a `base.yaml` key — because adding a key would have moved the hash away from `78f1e3700da15aa1`
+and invalidated every artifact. The resolution was a hash-exempt path
+(`docs/PHASE14_OPTICAL_SAR_DECISIONS.md`, item 7):
+
+```
+SATQUERY_CROMA_USE_8_BIT (env)  →  croma.use_8_bit (config)  →  default True
+```
+
+It is inspectable at runtime via `CROMAEncoder.describe()["input_normalisation"]`. This is recorded
+because it is the one place where "frozen" and "configurable" were reconciled deliberately, and a
+reviewer should know the escape hatch exists and is documented rather than hidden.
+
+### 2.7 What invalidates the hash
+
+Anything that changes `self._data` when serialised with `sort_keys=True`. Concretely:
+
+- editing any value in `configs/base.yaml`;
+- adding or removing a key;
+- changing a value's *type* such that its `default=str` rendering changes.
+
+The consequence is not merely a different hash; it is that **every artifact keyed to
+`78f1e3700da15aa1` no longer describes the running configuration**, and the honest action is to
+re-run the evaluation rather than reinterpret the old numbers.
+
+---
+
+## 3. Reproduce the metric verification (cheap, no GPU)
+
+This is the cheapest, highest-value reproduction in the whole release. It takes seconds, needs no
+GPU and no network, and it is the check that a skeptical reviewer should run **first**.
+
+### 3.1 The verifier
 
 ```bash
 python release/tools/verify_readme_metrics.py
 ```
 
-- **Reads** the artifacts under `artifacts/`.
-- **Compares** each of the 20 quoted metrics at the precision printed in the README.
-- **Also asserts** statuses (VLM headline contains `ACCEPTANCE-REJECTED`; router `corpus_limited` /
-  `n_val`; calibration temperature and `ece_improvement`).
-- **Exits 0** and prints `ALL CLAIMS VERIFIED` only when everything matches.
+`release/tools/verify_readme_metrics.py` is read-only. Its own docstring states its contract:
+
+> Verify every metric quoted in the release README against its source artifact. Read-only. Each
+> claim is compared at the precision at which the README states it.
+
+The tool is **not** a re-computation of the metrics. It is a **provenance check**: it reads the
+artifact that the README cites, resolves a dotted key into it, and compares the value against the
+literal string the README prints — at the README's own precision.
+
+### 3.2 The 20 claims and their exact key paths
+
+The claim list is not assumed; the key names were **discovered by walking the artifacts**. Several
+live under nested paths, and the tool's docstring says so:
+
+> Key names were discovered by walking the artifacts, NOT assumed: several live under nested paths
+> (e.g. change metrics are `metrics.pooled.iou`, grounding is `results.head_threshold.mean_best_iou`,
+> VLM is `why_usable_verified.adapted_test.*`).
+
+| # | Claim | Artifact | Dotted key | README value |
+|---|---|---|---|---|
+| 1 | change pooled IoU | `artifacts/change/eval_test/eval_result.json` | `metrics.pooled.iou` | `0.8122` |
+| 2 | change macro IoU | `artifacts/change/eval_test/eval_result.json` | `metrics.macro.miou` | `0.8457` |
+| 3 | change pooled F1 | `artifacts/change/eval_test/eval_result.json` | `metrics.pooled.f1` | `0.8964` |
+| 4 | grounding canonical head_threshold mean_best_IoU | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json` | `results.head_threshold.mean_best_iou` | `0.2838` |
+| 5 | grounding canonical head_threshold recall@0.5 | `…/eval_result_canonical.json` | `results.head_threshold.recall.0.50` | `0.2198` |
+| 6 | grounding matched6 head_threshold mean_best_IoU | `…/eval_result_matched6.json` | `results.head_threshold.mean_best_iou` | `0.2566` |
+| 7 | grounding matched6 head_threshold recall@0.5 | `…/eval_result_matched6.json` | `results.head_threshold.recall.0.50` | `0.1938` |
+| 8 | grounding head_argmax mean_best_IoU (canonical) | `…/eval_result_canonical.json` | `results.head_argmax.mean_best_iou` | `0.1215` |
+| 9 | grounding zero-shot baseline IoU (canonical) | `…/eval_result_canonical.json` | `results.zero_shot_matched.mean_best_iou` | `0.0972` |
+| 10 | optical-SAR fusion accuracy | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` | `accuracy` | `0.931` |
+| 11 | optical-SAR fusion macro_F1 | `…/pre_registered_115_metric.json` | `macro_f1` | `0.434161` |
+| 12 | change_vqa test accuracy | `artifacts/change_vqa/run/PROMOTION.json` | `verification.test_accuracy` | `0.697626` |
+| 13 | change_vqa test macro_F1 | `artifacts/change_vqa/run/PROMOTION.json` | `verification.test_macro_f1` | `0.378373` |
+| 14 | change_vqa test2 accuracy | `artifacts/change_vqa/run/PROMOTION.json` | `verification.test2_accuracy` | `0.651469` |
+| 15 | change_vqa test2 macro_F1 | `artifacts/change_vqa/run/PROMOTION.json` | `verification.test2_macro_f1` | `0.372309` |
+| 16 | router overall ungated accuracy | `artifacts/router/threshold_sweep_val.json` | `overall_ungated_accuracy` | `0.965116` |
+| 17 | calibration ECE before scaling | `artifacts/calibration_v001.json` | `metrics.ece_before` | `0.013755` |
+| 18 | calibration ECE after scaling | `artifacts/calibration_v001.json` | `metrics.ece_after` | `0.014929` |
+| 19 | VLM adapter exact_match | `artifacts/vlm/phase6_closure.json` | `why_usable_verified.adapted_test.exact_match` | `0.963` |
+| 20 | VLM adapter F1 | `artifacts/vlm/phase6_closure.json` | `why_usable_verified.adapted_test.f1` | `0.96432` |
+
+Note rows 1–3, 6–7, 12–15, and 16: these are the rows that keep the honesty rules honest. The
+change numbers are quoted as **pooled and macro together** (rule 4); grounding is quoted under
+**both protocols** (rule 2); change-VQA is quoted under **both test sets** (rule 3); the router
+number is quoted with its **validation-only** caveat (rule 5). A verifier that checked only one
+number per task would be reproducible and *wrong*.
+
+### 3.3 The status assertions
+
+Metrics are not the only claims. After the 20 numeric comparisons, the tool asserts four statuses
+that the README also states (`verify_readme_metrics.py:117-131`):
+
+```
+VLM headline contains ACCEPTANCE-REJECTED : True
+VLM status                               : CLOSED
+router corpus_limited                    : True
+router n_val                             : 86
+calibration temperature (temperature_scaling.temperature) : 0.9772731820958189
+calibration ece_improvement              : -0.001174  (negative => calibration did NOT help)
+```
+
+These are the checks that prevent the documentation from quietly upgrading a status. In particular:
+
+- the **VLM** metric is usable but the artifact's own headline contains `ACCEPTANCE-REJECTED` — so
+  `USABLE ≠ ACCEPTED` (rule 7);
+- the **router** number is flagged `corpus_limited: true` with `n_val: 86` — so it can never be
+  presented as a test number (rule 5);
+- the **calibration** `ece_improvement` is **negative**, i.e. calibration made ECE *worse*, and the
+  tool prints that interpretation inline (rule 6).
+
+### 3.4 The nested-key resolution trick
+
+Two subtleties in the resolver are worth recording, because they are the kind of thing that turns a
+verifier into a **false-pass** machine if done naively (`verify_readme_metrics.py:69-90`):
+
+1. **Keys that themselves contain dots.** The grounding recall dictionary is keyed by the string
+   `"0.10"`, `"0.25"`, `"0.50"`. A naive `dotted.split(".")` walk would break
+   `results.head_threshold.recall.0.50` into `…recall → 0 → 50` and fail. The resolver therefore
+   tries the **longest matching key at each step first**, so `0.50` stays intact.
+2. **Precision matching.** The comparison is
+   `round(float(val), decimals(claimed)) == float(claimed)` — i.e. the artifact value is rounded to
+   the number of decimals the README prints, then compared. This is why row 15 shows the artifact
+   value `0.372308516` matching the README's `0.372309`: the README states six decimals and the
+   artifact rounds to it.
+
+A verifier that compared raw floats would report false differences on every rounded value. A
+verifier that split dots naively would report false differences on the recall rows. Both failure
+modes are **silent false failures** — the same class of bug discussed in §7.3 and §10.
+
+### 3.5 The committed output
+
+```
+STATUS   claim                                                  artifact       readme  source
+----------------------------------------------------------------------------------------------------------------------
+MATCH    change pooled IoU                                        0.8122       0.8122  artifacts/change/eval_test/eval_result.json#metrics.pooled.iou
+MATCH    change macro IoU                                         0.8457       0.8457  artifacts/change/eval_test/eval_result.json#metrics.macro.miou
+MATCH    change pooled F1                                         0.8964       0.8964  artifacts/change/eval_test/eval_result.json#metrics.pooled.f1
+MATCH    grounding canonical head_threshold mean_best_IoU         0.2838       0.2838  artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json#results.head_threshold.mean_best_iou
+...
+MATCH    VLM adapter exact_match                                   0.963        0.963  artifacts/vlm/phase6_closure.json#why_usable_verified.adapted_test.exact_match
+MATCH    VLM adapter F1                                          0.96432      0.96432  artifacts/vlm/phase6_closure.json#why_usable_verified.adapted_test.f1
+
+=== status assertions ===
+  VLM headline contains ACCEPTANCE-REJECTED : True
+  VLM status                               : CLOSED
+  router corpus_limited                    : True
+  router n_val                             : 86
+  calibration temperature (temperature_scaling.temperature) : 0.9772731820958189
+  calibration ece_improvement              : -0.001174  (negative => calibration did NOT help)
+
+RESULT: ALL CLAIMS VERIFIED
+```
+
+— `release/tools/readme_metrics_report.txt` (committed).
+
+The tool **exits 0** and prints `ALL CLAIMS VERIFIED` only when every numeric comparison is `MATCH`
+and no artifact is missing or lacks the key. Any `NOFILE`, `NOKEY`, or `DIFFER` line increments the
+failure counter, the final line reads `N CLAIM(S) FAILED`, and the process exits 1. This makes the
+check usable in CI: a non-zero exit is a hard failure.
+
+### 3.6 Why this is a real check and not a tautology
+
+A skeptic might object: "you wrote the README from the artifacts, so of course they match." The
+answer is that the check is not that the README was *derived* from the artifacts, but that the
+README's published claims remain **mechanically traceable** to files a third party also has. If
+someone edits the README to state a nicer number, the tool fails. If someone edits an artifact to a
+different number, the tool fails. The check binds the published prose to the recorded evidence, and
+it does so without trusting either.
+
+---
+
+## 4. Reproduce the environment
+
+### 4.1 What you need
+
+- **Python 3.11+** (`README.md` §Installation: "Python 3.11+ and a CPU are sufficient").
+- A **CPU is enough**. There is no CUDA requirement.
+- Network access on first use, to fetch the pinned backbones from the Hugging Face Hub.
+
+### 4.2 The two install profiles
+
+`requirements.txt` documents two profiles explicitly:
+
+```text
+#   CPU (local dev / schema / geo / unit tests):
+#       pip install -r requirements.txt
+#   GPU (Kaggle T4x2 / HF ZeroGPU): torch is preinstalled on both.
+#       Do NOT pin torch here — Kaggle and HF ship their own builds.
+```
+
+— `requirements.txt` (header comment).
+
+The pins, with the contract each one carries (`requirements.txt`):
+
+| Package | Pin | Why |
+|---|---|---|
+| `numpy` | `>=1.26,<3` | core |
+| `pyyaml` | `>=6.0` | config loading |
+| `pydantic` | `>=2.6,<3` | schema |
+| `rasterio` | `>=1.3` | geospatial IO |
+| `pyproj` | `>=3.6` | CRS handling |
+| `opencv-python-headless` | `>=4.9` | image ops |
+| `transformers` | `>=4.52` | SmolVLM via `AutoModelForImageTextToText`; `AutoModelForVision2Seq` is deprecated (finding **C-2**) |
+| `open-clip-torch` | `>=2.24` | RemoteCLIP loaded via `pretrained=<path>` so `load_checkpoint()` runs its state-dict fixups (**C-4**) |
+| `sentence-transformers` | `>=2.7` | router embedding |
+| `peft` | `>=0.10` | LoRA adapter |
+| `huggingface_hub` | `>=0.23` | pin RemoteCLIP / SmolVLM / CROMA revisions |
+| `safetensors` | `>=0.4` | adapter serialisation |
+| `einops` | `>=0.7` | **required** by the vendored `use_croma.py` (`from einops import rearrange`) |
+| `gradio` | `>=4.44` | UI (frozen deployment target) |
+| `reportlab` | `>=4.1` | reporting |
+| `pytest` | `>=8.0` | tests |
+| `pytest-cov` | `>=5.0` | coverage |
+
+The comment block closes with an instruction a reviewer should heed: *"do not 'fix' by changing these
+pins"* — the pins encode verified contracts, not preferences.
+
+### 4.3 The `einops` lesson — a dependency no document declared
+
+`requirements.txt` carries an unusually explicit note about `einops`:
+
+> REQUIRED by the vendored `specialists/optical_sar/vendor/use_croma.py` (`from einops import
+> rearrange`). Found missing on 2026-09-18 by executing the vendored module: it raised
+> `ModuleNotFoundError`, so the CROMA path was blocked on a dependency that no document declared.
+> Not optional: without it the encoder cannot be imported at all.
 
-Committed output: `release/tools/readme_metrics_report.txt`.
+This is recorded verbatim because it is a reproducibility fact: a missing dependency that no
+document mentioned was found **by executing the code**, not by reading a spec. A third party
+building the environment will hit the same wall if they install from a partial list.
 
-## 3. Reproduce the environment (local run)
+### 4.4 Create the environment
 
 ```bash
 git clone https://github.com/Anish-lab-blip/SatQuery-AI
 cd SatQuery-AI
 python -m venv .venv
-source .venv/Scripts/activate      # Windows git-bash; .venv/bin/activate on Linux/macOS
+source .venv/Scripts/activate      # Windows git-bash; use .venv/bin/activate on Linux/macOS
 pip install -r requirements.txt
 ```
 
-Python 3.11+ and a CPU are sufficient. Device is selected via `SATQUERY_DEVICE`; all placement is
-`.to(device)`, never `.cuda()`. Backbones are fetched from the Hugging Face Hub on first use, pinned
-by revision.
+— `README.md` §Installation.
 
-## 4. Reproduce the tests
+> **Trap (Windows, recorded in §10.2).** In the authoring environment, `pytest` exists **only** in
+> the repository virtualenv. Invoking the system `pytest` fails or resolves to a different
+> interpreter. Always invoke the venv interpreter explicitly:
+> `.venv/Scripts/python.exe -m pytest …`.
 
-| Suite | Command | Expected |
+### 4.5 Backbones are fetched, pinned by revision, and never redistributed
+
+The four third-party backbones are pinned by `repo_id` **and** `revision` (`README.md`
+§Reproducibility item 2; `models/manifest.json` per-artifact `backbone` fields):
+
+| Model | Revision | Role |
 |---|---|---|
-| Frontend live-wiring | `pytest tests/unit/test_frontend_live_wiring.py` | **106 passed** |
-| Doc/frontend suite | `pytest` on the 5 doc/frontend files | **183 passed** |
-| Full unit suite | `pytest tests/unit` | 5–6 **environmental** failures (sandbox delete guard × 4, 1 ordering flake, 1 stale adapter test) — see [`EVALUATION.md`](EVALUATION.md) §4 |
+| `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | VQA + captioning backbone |
+| `chendelong/RemoteCLIP` | `bf1d8a3ccf2d` | remote-sensing grounding encoder |
+| `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | router embedding |
+| `antofuller/CROMA` | `0dd28e3d633b` | optical/SAR fusion encoder |
+
+**No backbone weights are redistributed** in this release (`release/HF_RELEASE_VERIFICATION.md §4`).
+The released artifacts are small trained modules *over* these frozen backbones; anyone using them
+must fetch the backbones too. Pinning by commit prefix (not by a floating tag like `main`) is what
+makes "the same backbone" reproducible months later.
+
+### 4.6 Device selection
+
+Device is selected via `SATQUERY_DEVICE`, and **all placement is `.to(device)`, never `.cuda()`**
+(`README.md` §Installation; `core/config.py:86-91`):
+
+```python
+@property
+def device_preference(self) -> str:
+    override = os.environ.get("SATQUERY_DEVICE")
+    if override:
+        return override
+    return "cuda" if _torch_cuda_available() else "cpu"
+```
+
+This is why a CPU-only reviewer can run everything except the training paths. The env override is
+read **without importing torch** in the deployment context (`docs/DEPLOYMENT.md §3.2`), so a
+misconfigured device cannot crash the service at import time.
+
+---
+
+## 5. Reproduce the tests
+
+### 5.1 The suites and their expected results
+
+| Suite | Command | Expected | Status |
+|---|---|---|---|
+| Frontend live-wiring | `python -m pytest tests/unit/test_frontend_live_wiring.py -q` | **106 passed** | `VERIFIED` |
+| Doc/frontend suite | `pytest` on the 5 doc/frontend files | **183 passed** | `VERIFIED` |
+| Full unit suite | `python -m pytest tests/unit` | **5–6 environmental/ordering failures**, rest pass; a re-run passes 137 | `MEASURED` |
+
+The three rows tell a story a single number would hide: the two **targeted** suites are clean, and
+the **full** suite has 5–6 failures that are environmental or ordering-related, not regressions. The
+next subsections reproduce each.
+
+### 5.2 Frontend live-wiring regression suite — 106 passed
+
+```bash
+python -m pytest tests/unit/test_frontend_live_wiring.py -q
+```
+
+— `README.md` §Local development.
+
+This suite is the regression net for the frontend↔backend wiring: the query box, the intent panel,
+the task tags, the run identifiers, and the evidence rendering. It is the suite that would catch a
+re-introduction of the router defect described in §6.4.
+
+**Expected: 106 passed.** Note the historical figure, because a reviewer may see both numbers:
+`docs/FINAL_DELIVERY_REPORT.md §7` records **94 passed** for this file, while
+`CURRENT_RELEASE_STATE.md:121` and `README.md` §The test suites record **106 passed** ("re-run this
+session"). The difference is the suite growing after the delivery report was written — the
+regression tests added for the router fix took this file from 100 → 106 tests
+(`DELIVERY_REPORT_2026-09-25.md`, regression-tests note). This document quotes **106**, the
+re-run-this-session figure, and cites both. (The exact count of tests *collected* in a single
+full-suite invocation is `UNKNOWN — not established from the available evidence`; the per-suite
+counts are what the evidence establishes.)
+
+### 5.3 Doc/frontend suite — 183 passed
+
+The documentation-and-frontend suite is **five files**, named explicitly in
+`docs/FINAL_DELIVERY_REPORT.md §7`:
+
+- `test_frontend_guide_doc`
+- `test_frontend_live_wiring`
+- `test_api_contract_doc`
+- `test_runbook_doc`
+- `test_deploy_config`
+
+Together they report **183 passed**. These are the checks that the docs' internal links resolve, the
+frontend assets are present, the API contract document matches the code, the runbook is consistent,
+and the deploy config is coherent.
+
+The same three-row test table is published in this release at `README.md` §The test suites
+(`README.md:1063-1074`), so a public reader can see the counts without the project-internal report.
+The underlying record is `docs/FINAL_DELIVERY_REPORT.md §7` (project-internal); the frontend count
+was re-run this session to **106** (`release/CURRENT_RELEASE_STATE.md:121`). See also
+[`EVALUATION.md`](EVALUATION.md) §7.2.
+
+### 5.4 Full `tests/unit` — the 5–6 failures, reported honestly
+
+Running the entire unit tree trips **5–6** failures. They are reported here rather than hidden, and
+they are classified by cause exactly as `docs/FINAL_DELIVERY_REPORT.md §7` records them:
+
+| # | Failure | Cause | Regression? |
+|---|---|---|---|
+| 1–4 | `test_safe_delete_shim` (×4) | the sandbox's bulk-**delete guard** (Windows verbatim-path behaviour) | **No** |
+| 5 | one ordering flake in the **router route test** | passes in isolation; order/collection-dependent | **No** |
+| 6 | one **stale adapter test** | asserts `optical_sar` absent when CROMA is *unshipped* — **CROMA is now shipped** | **No** |
+
+**Why these are not regressions.** They were isolated by a re-run of the affected files **together**,
+which passes **137** tests (`docs/FINAL_DELIVERY_REPORT.md §7`;
+[`EVALUATION.md`](EVALUATION.md) §7.3–7.4). The `test_safe_delete_shim` failures are Windows-specific
+verbatim-path behaviour that does not occur on the deployment host; the ordering flake disappears when
+the router route test runs in isolation; the stale adapter test encodes an assumption (CROMA not yet
+shipped) that the codebase has since outgrown. None of them is a behavioural defect in shipped code.
+
+> **Honesty note.** The precise full-suite *collected* count is
+> `UNKNOWN — not established from the available evidence`. What the evidence establishes is: the two
+> targeted suites pass (106, 183), the full suite has 5–6 environmental/ordering failures, and a
+> re-run of the affected files passes 137.
+
+### 5.5 The evidence-engine determinism suite
+
+The evidence engine (the component that turns specialist output into a *proof*) has its own suite —
+`tests/unit/test_evidence_engine.py` (73 tests) — covering **determinism** and **purity**: the same
+input produces the same evidence, and computing evidence does not mutate inputs. This is what makes
+the "evidence is independent of the verdict" property in §6.5 hold.
+
+### 5.6 Invocation traps
+
+- **Use the venv interpreter.** `.venv/Scripts/python.exe -m pytest …` on Windows.
+- **The full-suite run trips a bulk-delete guard** in the authoring sandbox (§10.3). Run the
+  targeted suites to avoid it.
+- **Do not pipe pytest through `grep`** in the authoring sandbox — output is block-buffered and a
+  killed pipeline swallows it (§10.6). Redirect to a file instead.
+
+---
+
+## 6. Reproduce a live run
+
+### 6.1 The deployed stack
+
+The live topology is **Cloudflare Pages → Render → outbound tunnel → GitHub Codespace**
+(`docs/DEPLOYMENT.md §1`; `README.md` §Deployment):
+
+| Layer | Role | Host |
+|---|---|---|
+| Cloudflare Pages | static frontend | `https://satquery.pages.dev` |
+| Render | orchestrator / API gateway (`/api/*`, CORS, wake flow) | `https://satquery-backend-m4yv.onrender.com` |
+| GitHub Codespace | FastAPI inference host, CPU, port 8000 | `potential-space-trout-r4ppw969w45j2pvvw` |
+
+Deployed revisions (VERIFIED):
 
-> **Environment note.** In the authoring sandbox the full suite trips a **bulk-delete guard** and
-> pytest is only installed in the repository virtualenv. Neither affects a normal user environment.
+| Component | Revision |
+|---|---|
+| Frontend | `2d7ae53b482d` |
+| Backend / orchestrator | `89d80eaddec5` |
+| Inference | `5a0936ace491` |
 
-## 5. Reproduce a live run
+> **Trap.** The deployment sources are **private** repositories, separate from this release. The
+> `deploy/` directory inside the monorepo working copy is **stale and untracked** — it is *not* the
+> deployed source (`docs/DEPLOYMENT.md §1`). Reproducing a live run means driving the deployed
+> endpoints, not rebuilding from `deploy/`.
 
-The deployed stack is reachable:
+### 6.2 Health and capabilities
 
 ```bash
 curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/health
 curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/capabilities
 ```
 
-`/api/capabilities` returns six tasks, all `available: true`. A live run requires the tunnel agent to
-be connected (`agent_connected:true`); if the Codespace is stopped, the request parks until the
-tunnel timeout. See [`DEPLOYMENT.md`](DEPLOYMENT.md) §5–6.
+The `/api/health` payload (recorded live, `docs/DEPLOYMENT.md §2`):
+
+```json
+{"status":"ok","service":"satquery-orchestrator",
+ "tunnel":{"agent_connected":true,"agent_id":"codespaces-fd1038","pending":0,"completed":97},
+ "config":{"codespace_name":"potential-space-trout-r4ppw969w45j2pvvw\n","codespace_port":8000,
+           "transport_mode":"auto","tunnel_timeout_s":150.0,"wake_timeout_s":120.0,
+           "upstream_timeout_s":90.0,"device":"cpu","has_github_token":true}}
+```
+
+`/api/capabilities` returns **six tasks, all `available: true`**. A live run requires
+`tunnel.agent_connected: true`; if the Codespace is stopped, the request parks until the tunnel
+timeout (`docs/DEPLOYMENT.md §5`).
+
+> **`--noproxy '*'` is not optional in the authoring sandbox** (§10.1). The sandbox proxy is dead;
+> without the flag, the request fails before reaching Render. In a normal environment the flag is
+> harmless.
+
+### 6.3 What was run, and how to re-run it
+
+The behavioural validation is **3 passes × 8 cases = 24 live runs**, each pass **8/8**, with **0
+mock nodes** and a live trace bar at **94.4444 %** (`.workbuddy-ai/scratch/live_validation/
+LIVE_VALIDATION_POSTFIX.md`). The eight cases exercise all six specialist tasks plus the two
+router-defect queries (B1, B2).
+
+| pass | HEAD | harness | result |
+|---|---|---|---|
+| 1 | `ff46eba42b18`+`d413d3672311` | v1 (`fill_input`) | 8/8 |
+| 2 | `2d7ae53b482d` | v2 asserting | 8/8 |
+| 3 | `2d7ae53b482d` | v2 asserting (pre-discriminator-fix) | 8/8 (recomputed) |
+
+To re-run: drive `https://satquery.pages.dev` in a **headed** browser (the headed requirement is
+explained in §6.4), upload **one asset per case** for single-image tasks and a **pair** for the
+temporal tasks, submit the query, and read the run identifier, the `mock_nodes` count, the trace
+fill, and the answer's `[task]` tag. The eight expected cases are tabulated in
+[`EVALUATION.md`](EVALUATION.md) §6.2.
+
+### 6.4 The Chrome focus trap — why the harness asserts
+
+Pass 1's harness drove the query box with `fill_input()`, which types with **real CDP key events**.
+A re-run attempt failed on case 1 with `run_id=0002`, `mock_nodes=9`, `answer="No answer yet"`, and
+only a `capabilities` call — the **mock path**. Root cause:
+
+> Chrome **drops synthesized key events when the browser window does not hold OS focus**; the harness
+> had **no assertion**, so it clicked Run with the page's *default* query still in the box.
+
+Measured directly: with Chrome backgrounded, `press_key("Z")` left `#qtext.value` unchanged, while
+`type_text("Q")` (CDP `Input.insertText`, not focus-gated) inserted fine
+(`LIVE_VALIDATION_POSTFIX.md`, "Why this pass needed a new harness").
+
+This is the single most important trap for anyone reproducing a live run, because it produces a
+**silent false pass** — the pipeline "works", it just answered a different question. The fix was to
+**assert the input state before dispatching**, with three pre-dispatch assertions:
+
+- `q_ok` — the query box really held the query;
+- `obs_ok` — `#obsTail == 'ready'`;
+- `t0_ok` — both frames present where required.
+
+The full account, including the check that pass 1 was **not** infected, is in
+[`EVALUATION.md`](EVALUATION.md) §6.4–6.5. In short: pass 1's intents are query-specific (A1 reads
+`taskvqa…temporalnone`, not the default's `taskchange…temporalrequired`), their answers embed the
+query text, and A6's answer proves two files were uploaded — so pass 1's results are clean.
+
+### 6.5 The recompute-verdicts safety property
+
+Pass 3 was launched with a harness build that still carried two *discriminator* bugs (the answer
+`[task]` tag exists only for region tasks; the intent panel is a concatenated string needing a
+non-greedy match). Its raw output says `SUMMARY 0/8`; the verdicts are recomputed from the recorded
+evidence by `recompute_verdicts.py`, giving 8/8. This is not a workaround — it is the intended
+safety property:
+
+> The recorded evidence (run id, `mock_nodes`, intent, answer, assertions) is **independent of the
+> verdict computation**, so a harness bug can never silently turn a real failure into a pass, and
+> never costs a re-run to correct.
+
+— `LIVE_VALIDATION_POSTFIX.md`, Pass 3.
+
+Two further harness bugs were found and fixed, both causing **false failures** (not false passes):
+the answer `[task]` tag exists only for region tasks, and the intent panel needs a non-greedy match.
+The harness now computes the dispatched task as `answer_tag` when present, else the reading. Both
+are recorded because a false failure and a false pass are different risks and the project tracks
+them separately.
+
+### 6.6 Cold start and tunnel caveats
+
+- **Cold start.** Render's free tier sleeps and the Codespace may be stopped. The first request can
+  exceed the client timeout while weights are fetched; a retry a few seconds later normally
+  succeeds. Warm the stack before any demonstration and confirm `agent_connected: true`
+  (`README.md` §Deployment caveats).
+- **Tunnel gaps (B-07, `OPEN`).** The tunnel agent can be briefly absent; a request during a gap may
+  hang or return 504. **This is not fixed in production.** Root cause: in `auto` mode a tunnel
+  timeout falls through to the forwarded-port path, spending the 120 s wake timeout on a 302 — the
+  observed ~249 s failure (`docs/DEPLOYMENT.md §6`). A patch exists and was deliberately **not**
+  deployed.
+- **Never retry `POST /api/infer` at the gateway** — a retry consumes inference twice
+  (`docs/DEPLOYMENT.md §7`).
+
+---
+
+## 7. Reproduce and verify the released artifacts
+
+### 7.1 The manifest and checksums are generated, not typed
+
+`models/manifest.json` and `models/checksums.sha256` are produced by
+`release/tools/generate_model_manifest.py`, which reads the actual files:
+
+> Generated by reading the files. No byte count or hash is typed by hand. Backbones are NOT
+> redistributed; they are fetched from the Hugging Face Hub, pinned by revision.
+
+— `models/manifest.json` (`"note"` field).
+
+Verify the local weights:
+
+```bash
+sha256sum -c models/checksums.sha256
+```
+
+`models/checksums.sha256` (committed):
+
+```
+c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa  change/head.pt
+cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a  change_vqa/head.pt
+785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab  optical_sar/head.pt
+93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb  grounding/head.pt
+8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c  router/adapter.pt
+07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e  vlm/adapter_model.safetensors
+```
+
+The manifest records, per artifact: `id`, `task`, `kind`, `path` (in the project), `hf_path` (on the
+Hub), `backbone`, `architecture`, `source_metric_artifact`, `config_hash`, `bytes`, `sha256`, and
+`status`. The manifest's own header records `config_hash: 78f1e3700da15aa1` and
+`artifact_count: 6`, generated `2026-09-25T18:15:38+00:00`.
+
+### 7.2 Independent Hub verification
+
+```bash
+export HF_TOKEN=...            # token with repo.content.read
+python release/tools/hf_verify.py
+```
+
+`release/tools/hf_verify.py` is an **independent** check: it does not trust the upload step. It
+reads the manifest, **re-downloads each artifact from the Hub over direct HTTPS**, hashes the bytes
+it received, and compares against the locally-computed sha256. Its docstring is explicit that it
+uses direct HTTPS *deliberately*:
+
+> Note: `hf_hub_download` is deliberately NOT used here — in this environment it returned an empty
+> file (sha256 e3b0c442…), which would have produced a false FAIL. Direct HTTPS is the honest check.
+
+Recorded result (`release/tools/hf_verify_report.txt`):
+
+```
+repo            : thundercode/SatQuery
+private         : False
+sha (HEAD)      : 55681e0cddb91a4a5655da98a49bc025e537b657
+lastModified    : 2026-09-25T18:21:52.000Z
+files on Hub    : 22
+
+STATUS  hf_path                                    remote bytes   local bytes
+----------------------------------------------------------------------------------
+MATCH   change/head.pt                               63,231,009    63,231,009
+MATCH   change_vqa/head.pt                            5,822,809     5,822,809
+MATCH   optical_sar/head.pt                          14,427,457    14,427,457
+MATCH   grounding/head.pt                            12,639,041    12,639,041
+MATCH   router/adapter.pt                               211,961       211,961
+MATCH   vlm/adapter_model.safetensors                34,798,048    34,798,048
+
+artifacts verified : 6
+artifacts failed   : 0
+
+  OK  README.md
+  OK  MODEL_CARD.md
+  OK  models/manifest.json
+  OK  models/checksums.sha256
+```
+
+Total released weight payload: **131,130,325 bytes (~125 MiB)** across the six artifacts
+(`release/HF_RELEASE_VERIFICATION.md §4`). Exit code 0 with `artifacts failed : 0` means the release
+is intact.
+
+### 7.3 A verification method that was itself wrong — the empty-file false FAIL
+
+This is the most instructive reproducibility incident in the release, and it is recorded because a
+verifier that silently hashes an **empty file** is a verifier that can lie in *both* directions.
+
+The **first** verification attempt reported all six artifacts `DIFFER`, with every remote hash equal
+to:
+
+```
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+```
+
+— the sha256 of **empty content**. The cause was **not** the upload: `hf_hub_download` returned an
+empty file in the authoring environment (a download-path problem), so the verifier hashed nothing.
+
+It was caught by a **second, independent method** — a direct `curl` download — which produced the
+correct hash `8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` for
+`router/adapter.pt`, byte-identical to the local file and confirmed to be a real PyTorch zip
+(`PK\x03\x04`, containing `adapter/data.pkl`). The verifier was then rewritten to use direct HTTPS
+with proxies disabled (`ProxyHandler({})`), and now reports 6/6 `MATCH`.
+
+`release/HF_RELEASE_VERIFICATION.md §5.1` states the lesson directly:
+
+> The failed first attempt is recorded because a verifier that silently hashes an empty file would
+> have produced a **false failure** — and, with a different bug, could just as easily have produced
+> a **false pass**.
+
+**What a third party should take from this.** Do not use `hf_hub_download` as the *sole* verification
+path. The honest check is a direct HTTPS download with proxies disabled, hashing the bytes received,
+and comparing sizes as well as hashes. `hf_verify.py` does all three
+(`size_ok = remote_bytes == local_bytes == len(content)`; `hash_ok = remote_hash == local_hash`).
+
+### 7.4 Cross-checks against independently-recorded hashes
 
-## 6. Reproduce the trained artifacts
+Two of the six hashes can be checked against values recorded **elsewhere** in the project,
+independent of this release (`release/HF_RELEASE_VERIFICATION.md §6`):
 
-Backbones are public and pinned; the **trained modules** are released on the Hugging Face Hub under
-`thundercode/SatQuery`, each with its backbone dependency and checksum. To reproduce training:
+| Artifact | Recorded elsewhere | Computed here | Agree |
+|---|---|---|---|
+| `change_vqa_head` | `cfae5e43…d63a82a` (`artifacts/change_vqa/run/PROMOTION.json`) | `cfae5e43…d63a82a` | **yes** |
+| `vlm_lora_adapter` | `07c76a75…a5adf5e` (adapter provenance) | `07c76a75…a5adf5e` | **yes** |
+
+This is a stronger form of verification than self-consistency: it means the released bytes match
+hashes that were recorded by the training/export process, not merely by the release step.
+
+### 7.5 The evidence archive
+
+`release/tools/build_archive.py` builds the Phase-7 evidence archive. Per owner decision **G3**,
+everything under `artifacts/` is included **verbatim** — including duplicates and caches — so the
+archive is **~4 GB** and requires ZIP64. The layout:
+
+```
+SatQuery_AI_Final_Archive_2026-09-25/
+  README_ARCHIVE.md              (what is inside, what was excluded and why)
+  release/                       (the curated public release tree, verbatim)
+  artifacts/                     (VERBATIM, per owner decision G3)
+  evidence/
+    live_validation/             (3 validation passes + screenshots)
+    delivery/                    (delivery report + handoff)
+  verification/                  (state, manifests, verification reports)
+```
+
+Weights and caches are stored with `ZIP_STORED` (no recompression) to avoid wasting hours on
+already-compressed data. The archive **excludes** secret/token files, OS junk, virtualenvs,
+`node_modules`, the HF hub cache, and temp browser profiles — documented in `README_ARCHIVE.md`,
+never silent. It is verified by `release/tools/verify_archive.py`, which extracts to a separate temp
+directory and checks the CRC of every member plus the sha256 of every artifact against the live
+files.
+
+---
+
+## 8. What "reproduce" means for the two externally-trained artifacts
+
+Six trained artifacts are released. **Four were trained inside the project's own pipeline; two were
+trained outside this repository** on external GPU. This distinction is the heart of the
+reproducibility contract, so it is stated plainly rather than implied.
 
 | Artifact | Where it trains | Guide |
 |---|---|---|
-| router adapter | local CPU | `configs/base.yaml` §router.training |
-| grounding head | local | `configs/base.yaml` §grounding_training |
-| change head | local | `configs/base.yaml` §change |
-| optical_sar fusion head | local, seed sweep | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` |
-| change_vqa head | **external GPU (Kaggle)** | `docs/R02_KAGGLE_TRAINING_GUIDE.md` |
-| vlm LoRA adapter | **external GPU** | `configs/base.yaml` §training |
+| `router/adapter.pt` | local CPU | `configs/base.yaml` §router |
+| `grounding/head.pt` | local | `configs/base.yaml` §grounding_training |
+| `change/head.pt` | local | `configs/base.yaml` §change |
+| `optical_sar/head.pt` | local, seed sweep | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` |
+| `change_vqa/head.pt` | **external GPU (Kaggle)** | `docs/R02_KAGGLE_TRAINING_GUIDE.md` |
+| `vlm/adapter_model.safetensors` | **external GPU** | `configs/base.yaml` §training |
+
+### 8.1 What the repository reproduces for the external artifacts
+
+For the two externally-trained artifacts, this repository reproduces:
+
+1. **The promotion gate** — the checks the returned checkpoint must pass before it is accepted
+   (byte-identity, sha256 match against the recorded digest, zero non-finite tensors).
+2. **The evaluation** — the scoring path that turns a checkpoint into the recorded metrics.
+3. **The serving wiring** — the code path that loads the artifact and routes to it.
+
+It does **not** ship a one-command retrain for those two artifacts. That is stated, not implied.
+
+### 8.2 The change-VQA path (Kaggle)
+
+`docs/R02_KAGGLE_TRAINING_GUIDE.md` is the authoritative guide; it states its own scope:
+
+> **Audience:** the person running the external GPU job. **Scope:** everything needed to go from
+> this repository to a returned, reviewable checkpoint. **Not in scope:** deciding whether R-02 is
+> done — that happens after the artifact comes back.
+
+Its status is `IMPLEMENTATION_READY_FOR_EXTERNAL_TRAINING`, and it is explicit that *"training
+produces an artifact, not a verified capability, and the run record says `TRAINED_UNVERIFIED`."*
+
+The reproduction is **byte-scoped**: the upload set is defined by a digest so the external run can be
+shown to have started from exactly this code.
+
+| Item | Value |
+|---|---|
+| Upload set | **249 files**, at the root of the archive |
+| Digest-defining subset | **248 files, 67,225,247 bytes** |
+| Manifest hash | `e98d3854db5f3edec76c1748c0c1bc3041d006ae5858debee31701539af7d0d6` |
+| Archive | `satquery-ai.zip` — 250 entries, 60.0 MB compressed |
+| Inside the archive | `UPLOAD_MANIFEST.txt` — every file with size and SHA256, plus the digest |
+
+The digest covers **248 files, not 249**, for a principled reason the guide states: the guide itself
+lives in `docs/`, so it is part of the upload set — but it also *records* the digest, and a document
+cannot contain the hash of itself. The digest is therefore defined over the upload set **excluding
+this one file**; that is the only definition that can be reproduced. For the same reason the byte
+total for all 249 files is **not** quoted in the guide — it changes whenever the guide is edited; the
+authoritative value is in `UPLOAD_MANIFEST.txt`.
+
+The frozen STANet checkpoint the head depends on is verified by size and SHA256 in the notebook:
+
+```
+checkpoint : <CODE_ROOT>/artifacts/change/levir_change_v001/head.pt
+bytes      : 63,231,009
+sha256     : c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa
+checkpoint verified against the frozen digest
+```
+
+Training knobs are fixed, not tuned-to-pass: `HARD_STOP_SECONDS = 3 * 3600`, `EPOCHS = 40`,
+`BATCH_SIZE = 256`, `DEVICE = "cuda"`, `SEED = 42` (the trainer's own defaults are
+`epochs=40, batch_size=128, seed=42, patience=6, time_limit=10800s`). The guide instructs: *"Do not
+change these to get past an error."*
+
+Split integrity is measured, not assumed:
+
+```
+measured corpus ground truth (unique scenes / resolved questions):
+  Train    1600 scenes   65967 questions
+  Val       400 scenes   16441 questions
+  Test      968 scenes   39686 questions
+  Test2     968 scenes   31036 questions
+integrity clean : True
+scenes/split    : {'Train': 1600, 'Val': 400}
+overlap         : {'Train|Val': 0}
+```
 
-### 6.1 What "reproduce" means for externally-trained artifacts
+The two test sets (Test 39686, Test2 31036) are why change-VQA is quoted under **two** test sets
+(rule 3) — and why the promotion artifact carries both.
 
-The change-VQA and VLM artifacts were trained **outside this repository**. Reproducing them requires
-the external GPU environment described in their guides. The repository reproduces:
-- the **promotion gate** (byte-identity, sha256, zero non-finite tensors);
-- the **evaluation**;
-- the **serving wiring**.
+### 8.3 The VLM path
 
-It does **not** ship a one-command retrain for those two artifacts. This is stated rather than
-implied.
+The VLM artifact is a PEFT LoRA adapter (`r=16, alpha=32, dropout=0.05`) on the text-model
+projections of `HuggingFaceTB/SmolVLM-500M-Instruct` (`models/manifest.json`). Its acceptance status
+is **`ACCEPTANCE-REJECTED`** — the metrics are usable (`exact_match 0.963`, `f1 0.96432`) but the
+adapter was **not promoted**. `USABLE ≠ ACCEPTED` (rule 7). Reproducing this artifact's *training*
+requires the external GPU environment described in `configs/base.yaml` §training; reproducing its
+*evaluation* and its *serving wiring* is done by this repository. The rejection is preserved in the
+record (`artifacts/vlm/phase6_closure.json`), not scrubbed.
 
-## 7. What is NOT reproducible from this release
+### 8.4 The optical-SAR seed sweep
+
+The optical-SAR head was trained locally with a **seed sweep**; the selected production artifact is
+`fusion_head_production_v001`, and its pre-registered metric is the 11.5 target recorded in
+`docs/PHASE12_115_METRIC_COMPUTED.md` and `artifacts/optical_sar/fusion_head_production_v001/
+pre_registered_115_metric.json`. The input dimension is **derived at runtime** (not hardcoded) —
+`2318 = 3×768 + 12 + 2` (`fusion_head.py:74`, `:311`), independently confirmed in
+`docs/PHASE14_OPTICAL_SAR_DECISIONS.md §1`. The ruling is **`OPEN`**, and the accuracy (`0.931`) is
+never quoted without the macro-F1 (`0.434161`) — rare classes are poorly handled (rule 4).
+
+### 8.5 What is *not* one-command
+
+There is no script that takes this repository and produces all six artifacts in one invocation. The
+four locally-trained artifacts have local training paths, and the two external ones require an
+external GPU environment. This is a deliberate boundary: the release ships **frozen artifacts with
+provenance**, not a retraining harness.
+
+---
+
+## 9. What is NOT reproducible from this release — exhaustive
+
+This section is the honest counterweight to §1. Every item below is something a reader might
+reasonably expect to reproduce and **cannot** from this release alone.
+
+### 9.1 Not reproducible because the source is not released
 
 | Item | Reason |
 |---|---|
-| The private deployment repos | They are private; the deployed sources are not in this release. |
-| System-level end-to-end benchmark | **No such benchmark exists.** |
-| Router test-split number | **Not run.** |
-| CDVQA / SECOND imagery | Public but large; the release documents the acquisition + name-verification procedure, not the data. |
-| BigEarthNet full corpus | **Not downloaded** (only a 28k S2 subset was used). |
-| The historical ZeroGPU/Gradio deploy target | Frozen paperwork only; no runtime exists in code. |
-
-## 8. Environment traps recorded for reproducibility
-
-- **Dead proxy in the authoring sandbox** — outbound calls need `--noproxy '*'` (curl) or
-  `ProxyHandler({})` (Python).
-- **pytest only in the repo virtualenv** (`.venv/Scripts/python.exe`).
-- **Full-suite pytest trips a bulk-delete guard** in the sandbox.
-- **Cloudflare 308-redirects `X.html` → `/X`.**
-- **Chrome drops synthetic CDP key events when the window lacks OS focus** — relevant to any
-  browser-driven reproduction of the live validation.
+| The three **private deployment repositories** (`SatQuery-Frontend`, `SatQuery-Backend`, `SatQuery-Inference`) | Private by design; their links 404 for an outside audience (`docs/DEPLOYMENT.md §6`). |
+| The **live deployment** itself | Requires the private sources plus the Render/Codespace/Cloudflare accounts. |
+| The **monorepo working copy** | Local only, no remote; 334 dirty entries (`docs/DEPLOYMENT.md §1`). |
+
+### 9.2 Not reproducible because it does not exist or was not run
+
+| Item | Reason |
+|---|---|
+| **System-level end-to-end benchmark** | **No such benchmark exists.** Per-specialist metrics are real; a single end-to-end number is `NOT RUN` (`README.md` §Known limitations 1). |
+| **Router test-split number** | **The test split was never run.** The only router number is validation, ungated, `n = 86` (rule 5). |
+| **Captioning benchmark** | `IMPLEMENTED` but the benchmark is `NOT RUN` ([`EVALUATION.md`](EVALUATION.md) §4.8). |
+| **A composite / "overall accuracy"** | Forbidden by construction: `official_aggregate_weights: null`; `normalize.py` raises on aggregate requests (rule 8). |
+
+### 9.3 Not reproducible because the data is large or external
+
+| Item | Reason |
+|---|---|
+| **CDVQA / SECOND imagery** | Public but large. The release documents the acquisition + name-verification procedure, not the data. |
+| **BigEarthNet full corpus** | **Not downloaded** — only a 28k S2 subset was used. |
+| **LEVIR-CD-256 / VRSBench raw data** | Not redistributed; the splits (7120/1024/2048, and VRSBench `n=16159`) are documented, not shipped. |
+| **Fusion held-out test (n=4000)** | Documented, not shipped. |
+| **Feature caches** | Reproducible but included only in the verbatim archive, not in the release. |
+
+### 9.4 Not reproducible because it was external-GPU training
+
+| Item | Reason |
+|---|---|
+| **change-VQA retrain** | External GPU (Kaggle); the repo reproduces the *promotion gate*, *evaluation*, and *serving wiring*, not a one-command retrain (§8.2). |
+| **VLM LoRA retrain** | External GPU; and the adapter is `ACCEPTANCE-REJECTED` anyway (§8.3). |
+
+### 9.5 Not reproducible because it is frozen paperwork
+
+| Item | Reason |
+|---|---|
+| **The historical ZeroGPU / Gradio deploy target** | `configs/deploy.yaml` describes the old HF-Space/ZeroGPU target and is left undisturbed as frozen paperwork; **no Gradio runtime exists in code** (`docs/DEPLOYMENT.md §9`). Editing it would move the config hash. |
+
+### 9.6 Not reproducible *as a bit-identical training run*
+
+The released **artifacts** are byte-identical (guaranteed by hash). The **training runs** that
+produced them are not claimed to be bit-reproducible on arbitrary hardware: floating-point
+non-determinism, different accelerators, and library-version drift all affect a training run. What is
+guaranteed is: the *config* is frozen, the *seed* is recorded (`42`), and the *artifact* is hashable.
+
+### 9.7 Explicit non-claims
+
+To remove any ambiguity, this release does **not** claim:
+
+- that the live system achieves any single accuracy number;
+- that the router's `0.965116` is a test result (it is validation-only, `n = 86`);
+- that the VLM adapter is accepted (it is rejected);
+- that calibration improves ECE (it worsens it, `0.013755 → 0.014929`);
+- that optical-SAR fusion is production-ready (ruling `OPEN`, macro-F1 `0.434161`);
+- that change-VQA is settled (ruling `OPEN`, two test sets);
+- that grounding is solved (`0.2838` canonical IoU — useful, not solved);
+- that B-07 tunnel gaps are fixed (patch prepared, `NOT deployed`).
+
+---
+
+## 10. Environment traps recorded for reproducibility
+
+These are the edges that cost real debugging time in the authoring environment. They are recorded so
+a third party does not rediscover them — and, more importantly, so that a reviewer can distinguish
+"this is a documented environment quirk" from "this is a defect in the released code".
+
+### 10.1 The dead sandbox proxy needs `--noproxy '*'` / `ProxyHandler({})`
+
+**Symptom.** Outbound HTTP calls fail or hang in the authoring sandbox.
+**Root cause.** The sandbox proxy is dead; requests are routed to it and never reach the target.
+**Fix.** Disable proxies for the call:
+
+```bash
+curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/health
+```
+
+```python
+opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
+```
+
+`release/tools/hf_verify.py:37-38` does exactly this (`opener_no_proxy()`). In a normal environment
+the flag is harmless; in the sandbox it is mandatory.
+
+### 10.2 pytest exists only in the repository virtualenv
+
+**Symptom.** Invoking the system `pytest` fails or resolves to a different interpreter.
+**Root cause.** pytest is installed only in `.venv`.
+**Fix.** Always invoke the venv interpreter explicitly:
+
+```bash
+.venv/Scripts/python.exe -m pytest tests/unit/test_frontend_live_wiring.py -q
+```
+
+### 10.3 The full-suite pytest trips a bulk-delete guard
+
+**Symptom.** Running the whole `tests/unit` tree trips 4× `test_safe_delete_shim` failures.
+**Root cause.** Windows **verbatim-path** defects in the sandbox's bulk-delete guard. The precise
+condition under which the shim intermittently triggers on Windows is
+`UNKNOWN — not established from the available evidence`.
+**Fix / workaround.** Run the targeted suites (106 and 183 pass cleanly); treat the 4 shim failures
+as environmental, not regressions (§5.4).
+
+### 10.4 Cloudflare 308-redirects `X.html` → `/X`
+
+**Symptom.** A request to `something.html` returns a 308 redirect.
+**Root cause.** Cloudflare Pages 308-redirects `X.html` → `/X`.
+**Fix.** Reference the **extensionless** path (`docs/DEPLOYMENT.md §7`).
+
+### 10.5 Chrome drops synthetic CDP key events without OS focus
+
+**Symptom.** A browser-driven run silently answers the *default* query; the query box looks
+untouched; `mock_nodes` is non-zero.
+**Root cause.** Chrome **drops synthesized key events when the browser window does not hold OS
+focus**. `press_key`/`fill_input` (real CDP key events) are focus-gated; `Input.insertText`
+(`type_text`) is not.
+**Fix.** Use `type_text` (not `fill_input`), and **assert the input state before dispatch**
+(`q_ok`/`obs_ok`/`t0_ok`). This is the single most dangerous trap because it produces a **silent
+false pass** (§6.4; [`EVALUATION.md`](EVALUATION.md) §6.4).
+
+### 10.6 Buffering hides liveness (two separate causes)
+
+**Symptom 1.** A redirected browser-harness log stays at 0 bytes until the process exits —
+indistinguishable from a stall.
+**Root cause 1.** `browser-use` block-buffers stdout even when redirected.
+**Fix 1.** Force line-buffering and print a per-case marker (`CASE_START <id>`).
+
+**Symptom 2.** Piping the harness through `grep` swallows all output if the pipeline is killed.
+**Root cause 2.** `grep` block-buffers when piped.
+**Fix 2.** Redirect to a file instead of piping through `grep`.
+
+### 10.7 `hf_hub_download` returned an EMPTY file → a false FAIL
+
+**Symptom.** Every artifact hashes to `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`
+(empty content) and the verification reports all `DIFFER`.
+**Root cause.** `hf_hub_download` returned an empty file in the authoring environment (a download-path
+problem), so the verifier hashed nothing. The upload was fine.
+**Fix.** Use a **direct HTTPS download with proxies disabled**, hash the bytes received, and compare
+**size as well as hash**. `release/tools/hf_verify.py` is the honest check (§7.3).
+
+### 10.8 `codespace_name` carries a trailing `\n`
+
+**Symptom.** `/api/health` reports `"codespace_name":"potential-space-trout-r4ppw969w45j2pvvw\n"`.
+**Root cause.** B-02; the wake path strips it, so it is **cosmetic**.
+**State.** `OPEN (cosmetic)`; the fix is not deployed (`docs/DEPLOYMENT.md §2, §6`).
+
+### 10.9 A forwarded Codespace port returns 302 for a private repo
+
+**Symptom.** A forwarded Codespace port returns HTTP 302 instead of serving.
+**Root cause.** Private-repo port forwarding returns 302 — which is *why* the outbound tunnel exists
+(`docs/DEPLOYMENT.md §7`).
+**Fix.** Use the tunnel, not the forwarded port.
+
+### 10.10 The tunnel agent must be started by the devcontainer
+
+**Symptom.** A restarted Codespace comes up with `agent_connected: false`.
+**Root cause.** The tunnel agent is started by the devcontainer `postStartCommand`; if it does not
+run, the agent is absent.
+**Fix.** Ensure `.devcontainer/` starts the agent on start (`docs/DEPLOYMENT.md §7`).
+
+---
+
+## 11. What is NOT RUN / OPEN / BLOCKED for this topic
+
+Per the documentation standard, this section lists, for the topic of reproducibility itself, what is
+not settled.
+
+| Item | Status | Note |
+|---|---|---|
+| End-to-end system benchmark | **NOT RUN** | No such benchmark exists; nothing to reproduce. |
+| Router **test**-split number | **NOT RUN** | Only validation (`n = 86`, ungated) exists. |
+| Captioning benchmark | **NOT RUN** | Implemented; not benchmarked. |
+| VLM adapter acceptance | **REJECTED** | Metrics usable; not promoted. `USABLE ≠ ACCEPTED`. |
+| Optical-SAR fusion ruling | **OPEN** | Accuracy without macro-F1 is never quoted. |
+| Change-VQA ruling | **OPEN** | Two test sets. |
+| Grounding acceptance | **OPEN** | Two protocols; modest IoU. |
+| Calibration improvement | **REJECTED (retained in frozen config)** | ECE worsened `0.013755 → 0.014929`. |
+| B-07 tunnel gaps | **OPEN** | Patch prepared, **NOT deployed**. |
+| B-02 trailing `\n` | **OPEN (cosmetic)** | Not deployed. |
+| Licence | **OPEN** | **No LICENSE file exists**; owner decision. |
+| Exact full-suite *collected* count | **UNKNOWN** | Per-suite counts are known (106, 183, 137 re-run); the single collected total is not established. |
+| Exact Windows trigger for the delete-shim flake | **UNKNOWN** | The intermittent condition is not established from the available evidence. |
+| Bit-identical training on arbitrary hardware | **NOT CLAIMED** | Only artifacts are guaranteed byte-identical, by hash. |
+
+---
+
+## 12. Where the evidence lives
+
+| Evidence | Location |
+|---|---|
+| The 20-claim metric verifier | `release/tools/verify_readme_metrics.py` |
+| Its committed output | `release/tools/readme_metrics_report.txt` (`ALL CLAIMS VERIFIED`) |
+| The Hub verifier | `release/tools/hf_verify.py` |
+| Its committed output | `release/tools/hf_verify_report.txt` (6/6 `MATCH`) |
+| Hub release verification write-up | `release/HF_RELEASE_VERIFICATION.md` |
+| Model manifest (generated) | `models/manifest.json` |
+| Checksums (generated) | `models/checksums.sha256` |
+| Manifest generator | `release/tools/generate_model_manifest.py` |
+| Archive builder / verifier | `release/tools/build_archive.py`, `release/tools/verify_archive.py` |
+| Config loader + invariants | `core/config.py` (hash `:76-80`; `_validate` `:94`) |
+| Frozen config registry | `configs/base.yaml` (§evaluation `:273`, §deployment `:282`) |
+| Dependency manifest + profiles | `requirements.txt` |
+| Live topology, env vars, traps | `docs/DEPLOYMENT.md` |
+| Live validation (3 passes, 24 runs) | `.workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md`, `results_final.json`, `results_pass3.json`, `run_final2.txt`, `run_final3.txt` |
+| Verdict recomputation | `recompute_verdicts.py` |
+| Kaggle training guide (change-VQA) | `docs/R02_KAGGLE_TRAINING_GUIDE.md` |
+| Optical-SAR decisions | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` |
+| Pre-registered 11.5 metric | `docs/PHASE12_115_METRIC_COMPUTED.md` |
+| Grounding resolution decision | `docs/PHASE7_RESOLUTION_DECISION.md` |
+| Delivery test results (94/183/5–6/137) | `docs/FINAL_DELIVERY_REPORT.md §7` |
+| Frontend suite re-run count (106) | `release/CURRENT_RELEASE_STATE.md:121`, `README.md` §The test suites |
+| Evaluation protocols and honesty rules | [`EVALUATION.md`](EVALUATION.md) |
+| Benchmarks and their statuses | [`BENCHMARKS.md`](BENCHMARKS.md) |
+| Model cards and artifact details | [`MODELS.md`](MODELS.md) |
+| Datasets and splits | [`DATASETS.md`](DATASETS.md) |
+| Training paths | [`TRAINING.md`](TRAINING.md) |
+| Known limitations | [`LIMITATIONS.md`](LIMITATIONS.md) |