release: add docs/DEVELOPMENT.md

docs/DEVELOPMENT.md

@@ -0,0 +1,1330 @@
+# Development Guide
+
+**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `ATTEMPTED` · `NOT RUN` · `BLOCKED` ·
+`DEFERRED` · `OPEN` · `RESOLVED` · `BY DESIGN`.
+
+This is the contributor-facing guide to the SatQuery AI codebase. It answers: *what do I need to
+install*; *how do I run it*; *what is each directory for*; *what rules will break the build if I break
+them*; *how do I add a component*; *how do I test*; and *which environment traps will cost me an hour if
+nobody tells me about them*.
+
+Every command, path, timeout and rule below comes from a file that was read. Where the evidence does not
+settle a question, the text says `UNKNOWN — not established from the available evidence` rather than
+guessing.
+
+> **Read this first.** `configs/base.yaml` is the single registry, and its hash — **`78f1e3700da15aa1`**
+> — is frozen. **Editing the config moves the hash and invalidates every artifact keyed to it.** This is
+> the one rule in the repository whose violation is not locally reversible (§7).
+
+---
+
+## Table of contents
+
+**Part I — Prerequisites and setup**
+1. Prerequisites
+2. Local setup
+3. Running the service locally
+
+**Part II — Repository layout**
+4. The top-level directory map
+5. Where the contracts live
+6. The package layout inside `core/`, `specialists/`, `app/`
+
+**Part III — The configuration discipline**
+7. "No magic numbers in Python"
+8. What happens if you break the rule
+9. The hash-exempt path: environment overrides
+10. The invariants the loader enforces
+
+**Part IV — Contract-first development**
+11. `core/schemas.py` is the binding contract
+12. The specialist interface
+13. How to add a new specialist
+14. How evidence is emitted
+
+**Part V — The test workflow**
+15. Running tests
+16. The evidence-class markers
+17. The known failures, and why they are not regressions
+
+**Part VI — Scripts and notebooks**
+18. The class of scripts under `scripts/`
+19. `notebooks/`
+
+**Part VII — Training entry points**
+20. Local training (router, grounding, change, optical-SAR)
+21. External GPU training (change-VQA, VLM LoRA)
+22. Calibration
+
+**Part VIII — Coding conventions**
+23. What the code actually does
+
+**Part IX — Known development traps**
+24. The traps, in one table
+25. Trap 1 — the stale untracked `deploy/`
+26. Trap 2 — the dead sandbox proxy
+27. Trap 3 — pytest only in the venv
+28. Trap 4 — Chrome drops synthetic CDP key events
+29. Trap 5 — Cloudflare `_headers` concatenate
+30. Trap 6 — the annotation-scope trap
+31. Trap 7 — the full-suite bulk-delete guard
+32. Trap 8 — the stale serve process
+
+**Part X — Status and evidence**
+33. `NOT RUN` / `OPEN` / `BLOCKED` / `UNKNOWN` for development
+34. Where the evidence lives
+
+---
+
+# Part I — Prerequisites and setup
+
+## 1. Prerequisites
+
+| Requirement | Value | Source |
+|---|---|---|
+| Python | **3.11+** | `release/repo/README.md` §Installation |
+| Device | **a CPU is sufficient**; no CUDA requirement | `release/repo/README.md` §Installation |
+| GPU | not required | `docs/DEPLOYMENT_DECISION.md` §5 |
+| Git | needed to clone | `release/repo/README.md` §Installation |
+
+The codebase is CPU-first and the deployment is CPU-only. This is not a fallback — it is the design:
+
+- `core/config.py:86-91` — `device_preference` honours the `SATQUERY_DEVICE` env override, else
+  `"cuda" if torch.cuda.is_available() else "cpu"`.
+- Every specialist defaults to `device="cpu"` (`docs/DEPLOYMENT_DECISION.md` §5).
+- **No `.cuda()` call exists anywhere**; all placement is `.to(device)` (`docs/DEPLOYMENT_DECISION.md`
+  §5).
+- `configs/base.yaml:293` — `cpu_mode_required: true`.
+
+The development container that runs the inference tier pins Python 3.12
+(`.devcontainer/devcontainer.json:3`, `image: mcr.microsoft.com/devcontainers/python:3.12`). The
+repository's own guidance is 3.11+; a 3.11 or 3.12 interpreter both work.
+
+**Not implemented** (noted, not needed): no thread capping (`torch.set_num_threads`) and no
+quantisation. `lazy_load` and `cache_max_models` are only *reported* (`app/deployment.py:609-610`,
+`:896-897`); **UNVERIFIED** whether any code enforces a one-model cache (`docs/DEPLOYMENT_DECISION.md`
+§5).
+
+## 2. Local setup
+
+```bash
+git clone https://github.com/Anish-lab-blip/SatQuery-AI
+cd SatQuery-AI
+python -m venv .venv
+source .venv/Scripts/activate      # Windows git-bash; use .venv/bin/activate on Linux/macOS
+pip install -r requirements.txt
+```
+
+(`release/repo/README.md` §Installation)
+
+### 2.1 The dependency manifest and its two profiles
+
+`requirements.txt` is **frozen at architecture v1.0** and declares two install profiles in its header
+comment (`requirements.txt:1-8`):
+
+```
+# Two install profiles:
+#   CPU (local dev / schema / geo / unit tests):
+#       pip install -r requirements.txt
+#   GPU (Kaggle T4x2 / HF ZeroGPU): torch is preinstalled on both.
+#       Do NOT pin torch here — Kaggle and HF ship their own builds.
+```
+
+**torch is deliberately not pinned** (`requirements.txt:21`). The platform supplies it. This is why the
+local CPU install works on a machine with no CUDA and why the Kaggle/ZeroGPU targets get their own
+builds.
+
+The manifest's sections, with the contract notes the file itself carries:
+
+| Section | Packages | Note |
+|---|---|---|
+| core | `numpy`, `pyyaml`, `pydantic` | — |
+| geospatial | `rasterio`, `pyproj`, `opencv-python-headless` | — |
+| models | `transformers>=4.52`, `open-clip-torch>=2.24`, `sentence-transformers>=2.7`, `peft>=0.10`, `huggingface_hub>=0.23`, `safetensors>=0.4`, `einops>=0.7` | see below |
+| ui + reporting | `gradio>=4.44`, `reportlab>=4.1` | — |
+| dev | `pytest>=8.0`, `pytest-cov>=5.0` | — |
+
+Three contract notes in the file are load-bearing (`requirements.txt:43-53`):
+
+- `transformers>=4.52` — SmolVLM via `AutoModelForImageTextToText`; `AutoModelForVision2Seq` is
+  deprecated (finding C-2).
+- `open-clip-torch` — the RemoteCLIP checkpoint is loaded via `pretrained=<path>` so
+  `load_checkpoint()` runs its state-dict fixups (C-4).
+- `huggingface_hub` — used to pin the RemoteCLIP / SmolVLM / CROMA revisions.
+
+> **The `einops` lesson.** `einops>=0.7` is **not optional**. It is required by the vendored
+> `specialists/optical_sar/vendor/use_croma.py` (`from einops import rearrange`). It was found missing on
+> 2026-09-18 by executing the vendored module: it raised `ModuleNotFoundError`, so the CROMA path was
+> blocked on a dependency that no document declared (`requirements.txt:28-33`).
+
+### 2.2 Device selection
+
+Set `SATQUERY_DEVICE` to force a device:
+
+```bash
+export SATQUERY_DEVICE=cpu        # cpu | cuda | mps | null
+```
+
+`device_preference` reads the env var first and only falls back to a torch probe if it is unset
+(`core/config.py:87-91`). The value is read **without importing torch** on the metadata path, which is
+what keeps the health route cheap (`docs/DEPLOYMENT_TOPOLOGY.md` §3.3).
+
+## 3. Running the service locally
+
+The inference service is served by the same launcher the Codespace runs
+(`release/repo/README.md` §Local development):
+
+```bash
+# Inference service, CPU (this is the launcher the Codespace runs)
+PORT=8000 python deploy/codespace/serve.py
+
+# Health
+curl localhost:8000/v1/health
+```
+
+`serve.py` builds the app through `build_space_app()` and binds it with uvicorn on `$PORT` (default 8000)
+(`deploy/codespace/serve.py:16-25`). It imports cheaply — FastAPI is imported inside `build_space_app()`
+and no model is loaded at module scope — so it stays import-safe on a CPU host with no GPU and no weights
+present (`deploy/codespace/serve.py:1-13`).
+
+The server answers four routes (`app/space_app.py`):
+
+| Route | Line | Purpose |
+|---|---|---|
+| `GET /v1/health` | `app/space_app.py:521` | liveness, derived status, torch-free device probe |
+| `GET /v1/capabilities` | `app/space_app.py:549` | the six declared capabilities |
+| `POST /v1/assets` | `app/space_app.py:555` | handle-based upload (off unless enabled) |
+| `POST /v1/analyze` | `app/space_app.py:661` | the analysis path |
+
+`POST /v1/assets` is **off unless explicitly enabled** (`SATQUERY_ASSET_ENABLED` **and**
+`SATQUERY_ASSET_DIR` must both be set; otherwise it answers `503` rather than defaulting to a temp
+directory) (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §3.1.1).
+
+### 3.1 Serving the frontend locally
+
+The frontend is fully static and needs no build step (`release/repo/README.md` §Local development):
+
+```bash
+python -m http.server 5500 --directory frontend
+```
+
+The gateway's dev-origin allowlist includes `localhost` and `127.0.0.1` on ports 3000, 5500, 5173, 8000
+and 8080 (`deploy/render/main.py:139-143`), so a local dev server on any of those ports is accepted
+without an env-var change.
+
+### 3.2 Running the gateway locally
+
+```bash
+PORT=10000 \
+GITHUB_TOKEN=<token> \
+CODESPACE_NAME=<codespace> \
+SATQUERY_ALLOWED_ORIGINS="https://satquery.pages.dev" \
+uvicorn deploy.render.main:app --port 10000
+```
+
+(`deploy/render/README.md` §Running locally — the placeholder is the repository's own.)
+
+Then exercise it:
+
+```bash
+curl http://localhost:10000/api/health
+curl -X POST http://localhost:10000/api/infer -H 'content-type: application/json' -d '{"query":"..."}'
+curl http://localhost:10000/api/capabilities
+```
+
+> **The monorepo `deploy/` is stale** (§25). Use it for local experimentation only; it is **not** the
+> deployed source and it lacks the tunnel code the live service runs.
+
+---
+
+# Part II — Repository layout
+
+## 4. The top-level directory map
+
+| Directory / file | Purpose | Source |
+|---|---|---|
+| `app/` | the serving composition root and the FastAPI entrypoint | `app/serving.py`, `app/space_app.py`, `app/deployment.py` |
+| `core/` | the config loader, the typed schemas, the controller, the planner, the registry, the error taxonomy | `core/config.py`, `core/schemas.py`, `core/controller.py`, `core/planner.py`, `core/registry.py`, `core/errors.py`, `core/code_revision.py` |
+| `router/` | the MiniLM intent router: encoder, classifier, adapter, training, label space, lexical fallback | `router/encoder.py`, `router/classifier.py`, `router/adapter.py`, `router/train.py`, `router/dataset.py`, `router/fallback.py`, `router/label_space.py` |
+| `specialists/` | the six specialist implementations behind one interface | `specialists/base.py`, `specialists/vqa/`, `specialists/grounding/`, `specialists/change/`, `specialists/optical_sar/` |
+| `preprocessing/` | raster loading, imagery, quality checks | `preprocessing/raster.py`, `preprocessing/imagery.py`, `preprocessing/quality.py` |
+| `geospatial/` | CRS handling and transforms | `geospatial/crs.py`, `geospatial/transform.py` |
+| `evidence/` | the evidence engine and confidence | `evidence/engine.py`, `evidence/confidence.py` |
+| `evaluation/` | manifests, leakage, metrics, normalisation, the runner, benchmark adapters, the frozen prompt set | `evaluation/manifests.py`, `evaluation/leakage.py`, `evaluation/metrics/`, `evaluation/normalize.py`, `evaluation/runner.py`, `evaluation/benchmark_adapters/`, `evaluation/prompt_freeze.json`, `evaluation/manifest_freeze.json` |
+| `training/` | the training loops and data adapters, split by task | `training/router/`, `training/grounding/`, `training/change/`, `training/change_vqa/`, `training/fusion/`, `training/vlm/`, `training/calibration/`, `training/data/` |
+| `configs/` | the frozen registry and the frozen deploy manifest | `configs/base.yaml`, `configs/deploy.yaml` |
+| `gateway/` | the standalone gateway: policy (decisions) + app (HTTP plumbing) | `gateway/policy.py`, `gateway/app.py`, `gateway/assets.py` |
+| `deploy/` | the deployment launchers — **stale and untracked in the monorepo** | `deploy/codespace/`, `deploy/render/` (§25) |
+| `tests/` | the suites, split by concern and evidence class | `tests/unit/`, `tests/integration/`, `tests/routing/`, `tests/geospatial/`, `tests/leakage/`, `tests/model/`, `tests/e2e/` |
+| `scripts/` | the flat set of executable helpers | §18 |
+| `notebooks/` | the Kaggle notebooks | §19 |
+| `artifacts/` | trained heads, checkpoints, caches, evidence archives | ~3.7 GB total (`release/CURRENT_RELEASE_STATE.md` §3) |
+| `frontend/` | the static site | staged by `scripts/stage_pages.mjs` |
+| `docs/` | the project's own engineering records | ~60 files |
+| `hf/` | the Hugging Face project card / model cards | — |
+| `demo/`, `benchmark/`, `reports/`, `data/`, `logs/` | supporting material | — |
+
+> **`training/router/` is empty** in the working copy; the router's training lives in `router/train.py`
+> and `router/dataset.py` and is driven by `scripts/train_router.py` (§20).
+
+> **The repository has no `pyproject.toml` and no `setup.py`.** `app` is a plain package, so the repo root
+> must be on `sys.path` for `from app.space_app import ...` to resolve
+> (`deploy/codespace/launch.sh:31-34`; `pytest.ini` sets `pythonpath = .`).
+
+## 5. Where the contracts live
+
+Three files are authoritative, and the code — not a document — wins when they disagree:
+
+| Contract | File | What it fixes |
+|---|---|---|
+| the config registry | `configs/base.yaml` | every tunable value; hashed (§7) |
+| the wire/data schemas | `core/schemas.py` | the request, result, evidence, trace and health shapes |
+| the specialist interface | `specialists/base.py` | the four-method contract every specialist implements |
+| the capability table | `core/registry.py` | which capabilities exist and how to build them |
+| the planner's mapping | `core/planner.py` | `TASK_CAPABILITY` and `CAPABILITY_ASSETS` |
+
+When `docs/API_CONTRACT.md` and `core/schemas.py` disagree, the contract document's own authority clause
+resolves it: *"the request/response shapes are not invented here. They are the existing, tested Pydantic
+models in `core/schemas.py`. This document describes them; it does not declare new ones."*
+(`docs/STEP7_BACKEND_CHAIN_REPORT.md` §15). The known instance of this — the forward-compatibility
+promise versus `extra="forbid"` — is recorded as **C-2** and left as a documented contradiction rather
+than silently resolved (`docs/STEP7_BACKEND_CHAIN_REPORT.md` §15).
+
+## 6. The package layout inside `core/`, `specialists/`, `app/`
+
+### 6.1 `core/`
+
+| File | Role |
+|---|---|
+| `core/config.py` | the loader, the validator, the hash (§7, §10) |
+| `core/schemas.py` | every Pydantic model (§11) |
+| `core/controller.py` | the nine-state controller that runs a plan and produces the trace |
+| `core/planner.py` | the policy layer: `TASK_CAPABILITY`, `CAPABILITY_ASSETS`, `plan()` |
+| `core/registry.py` | the capability registry: spec table, lazy construction, degradation states |
+| `core/errors.py` | the typed error taxonomy and `scrub_paths` |
+| `core/code_revision.py` | the code revision recorded in a run |
+
+The three-tier control split is deliberate: the **registry** knows *which* specialists exist and *how* to
+construct them; the **planner** *decides* what runs; the **controller** executes. The registry's own
+docstring states it: *"It never decides what runs — that is `core.planner`'s job alone"*
+(`core/registry.py:6-8`).
+
+### 6.2 `specialists/`
+
+| Package | Files | Capability |
+|---|---|---|
+| `specialists/vqa/` | `inference.py`, `model.py`, `prompts.py` | `vqa`, `caption` |
+| `specialists/grounding/` | `specialist.py`, `remoteclip.py`, `head.py`, `inference.py` | `grounding` |
+| `specialists/change/` | `specialist.py`, `stanet.py`, `vqa_specialist.py`, `postprocess.py` | `change`, `change_vqa` |
+| `specialists/optical_sar/` | `specialist.py`, `croma.py`, `fusion_head.py`, `sensor_adapter.py`, `radiometry.py`, `inference.py`, `prompts.py`, `vendor/` | `optical_sar` |
+
+### 6.3 `app/`
+
+| File | Role |
+|---|---|
+| `app/serving.py` | the composition root — wires checkpoints through the registry's `builders=` override |
+| `app/space_app.py` | `build_space_app()` — the FastAPI app and the four routes |
+| `app/deployment.py` | the deployment description and device resolution |
+
+---
+
+# Part III — The configuration discipline
+
+## 7. "No magic numbers in Python"
+
+`configs/base.yaml` opens with the rule, in the file itself:
+
+```yaml
+# RULE: no magic numbers anywhere in Python. Everything tunable lives here.
+# Every value below is loaded, validated and hashed by core/config.py.
+```
+
+(`configs/base.yaml:4-5`)
+
+The loader reinforces it: *"One config system. No duplicated constants. Every value in configs/base.yaml
+is loaded, validated against the frozen architecture, and hashed so evaluation runs are reproducible."*
+(`core/config.py:1-5`).
+
+**Access pattern.** Never read the YAML directly; import the singleton:
+
+```python
+from core.config import get_config
+
+cfg = get_config()
+cfg.get("croma.image_resolution")     # dotted-path access
+cfg.require("change.encoder")         # raises ConfigError if missing
+cfg.seed                              # project.seed, default 42
+```
+
+`get_config()` is an `lru_cache(maxsize=1)` singleton — *"Import this, do not re-read YAML"*
+(`core/config.py:270-273`).
+
+## 8. What happens if you break the rule
+
+Two distinct failures, and both are loud.
+
+### 8.1 The loader fails startup
+
+`Config.__init__` calls `_validate()`, which collects **every** violation and raises a single
+`ConfigError` naming them all (`core/config.py:46-49,94-222`):
+
+```python
+if errors:
+    raise ConfigError(
+        "configuration failed frozen-architecture validation:\n  - "
+        + "\n  - ".join(errors)
+    )
+```
+
+The header states the intent: *"if a config tries to violate a frozen decision (e.g. bf16 on T4,
+torch.compile on ZeroGPU, a CROMA image_resolution that is not a multiple of 8), it fails loudly rather
+than at runtime"* (`core/config.py:6-8`).
+
+### 8.2 Editing the config MOVES the hash
+
+The hash is a sha256 over the whole registry, truncated to 16 hex characters
+(`core/config.py:76-80`):
+
+```python
+@property
+def hash(self) -> str:
+    """Stable hash of the whole registry. Recorded in every evaluation run."""
+    blob = json.dumps(self._data, sort_keys=True, default=str).encode()
+    return hashlib.sha256(blob).hexdigest()[:16]
+```
+
+Because it is computed over the entire registry, **any** edit to `configs/base.yaml` changes it. The
+frozen value is **`78f1e3700da15aa1`** (`release/repo/README.md` §Reproducibility;
+`docs/PHASE18_DEPLOYMENT_PACKAGING.md` §3). Every artifact records the hash it was produced against, so a
+hash change **invalidates every artifact keyed to it**.
+
+> **This is the one non-reversible action in the repository.** `docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §6.1
+> lists it as the single row in the rollback table whose answer to "Reversible?" is **no**: *"The recorded
+> benchmark hash is gone; the frozen benchmark no longer matches."*
+
+**Verify the hash after any config-adjacent change:**
+
+```bash
+$PY -c "from core.config import get_config; print(get_config().hash)"
+# expected: 78f1e3700da15aa1
+```
+
+(`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §1.1, where `$PY` is `$REPO/.venv/Scripts/python.exe` on Windows.)
+
+**Two things that do *not* move the hash** — both verified:
+
+1. **`configs/deploy.yaml` is inert.** It carries a `registry: false` marker and is never merged into the
+   registry, so editing it cannot move the hash. Verify with `$PY scripts/validate_deploy_config.py` →
+   exit 0, `"deploy.yaml is inert (not in the registry) and C-8-consistent."`
+   (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §2.4). **But** `scripts/validate_deploy_config.py` hard-fails if
+   the `deployment:` block in `deploy.yaml` differs key-for-key from `base.yaml`'s, so a change to one
+   alone fails the validator (`docs/DEPLOYMENT_DECISION.md` §4).
+2. **The serving wiring uses `builders=`.** `app/serving.py` wires the change and change-VQA heads through
+   the registry's `builders=` override instead of config, which is *"the seam that keeps `Config.hash`
+   unchanged while still pointing serving at the trained artifacts"* (`docs/PHASE19_FINAL_HARDENING.md`
+   §3.4).
+
+## 9. The hash-exempt path: environment overrides
+
+Two registry values can be overridden from the environment without editing the YAML
+(`core/config.py:261-265`):
+
+| Variable | Effect |
+|---|---|
+| `SATQUERY_PRECISION` | overrides `training.precision` |
+| `SATQUERY_TORCH_COMPILE` | overrides `deployment.torch_compile` (`"true"` → `True`) |
+
+**Both are still validated.** Setting `SATQUERY_TORCH_COMPILE=true` **fails startup** because finding C-8
+forbids `torch.compile` (`release/repo/docs/DEPLOYMENT.md` §6.3; `core/config.py:114-119`).
+
+This is the general pattern for deployment state that must not move the hash: read it from the
+environment. The asset-store capacity, TTL and per-file cap follow the same rule
+(`release/repo/README.md` §Installation).
+
+## 10. The invariants the loader enforces
+
+`core/config.py::_validate` is not documentation — it is a check that raises `ConfigError`. Each invariant
+exists because a specific finding proved the failure mode.
+
+| Invariant | Why it exists | Finding |
+|---|---|---|
+| `croma.image_resolution % 8 == 0` | CROMA asserts this; native 120 → 225 patches | C-7 |
+| `training.precision ∈ {fp16, bf16, fp32}` | the T4 is SM 7.5, so bf16 is unavailable | C-6 |
+| `deployment.torch_compile is not true` | ZeroGPU does not support `torch.compile` | C-8 |
+| `vlm.processor_longest_edge ≤ image.tile_size` | the processor's default `longest_edge` is 2048, which upscales a 512 px tile 4× and then splits it into **17** sub-images — a ~17× overrun, not the 4× the plan estimated | F5-2 |
+| `vlm.prompt_must_use_chat_template is true` | SmolVLM raises `ValueError` on prompts lacking one `<image>` token per image | F5-3 |
+| `fusion.input_dim == 3*encoder_dim + optical_channels + sar_channels` (= 2318) | CROMA emits optical/SAR/joint GAP vectors; the availability mask is consumed by the head | C-1 |
+| `croma.optical_channels == 12` and `croma.sar_channels == 2` | CROMA's `s2_channels` / `s1_channels` are fixed | — |
+| `grounding_head.feature_dim == 4 * grounding.encoder_projected_dim` (= 2048) | a mismatch is a **silent** shape error — torch raises only at the similarity step, after patch features are already cached | P7-1 |
+| `router.tasks` includes `unsupported` and `router.num_tasks == len(router.tasks)` | the ontology and its declared size cannot drift apart | — |
+| `change.sa_mode ∈ {BAM, PAM}` and `change.encoder` is set | the change architecture is not implicit | C-9 |
+| `image.top_k_tiles ≤ image.max_tiles` | the dispatch ceiling cannot exceed the examination ceiling | — |
+
+(`core/config.py:94-222`; `release/repo/README.md` §Installation.)
+
+> **Why the grounding-head guard matters most.** A mismatch there is a *silent* shape error: torch raises
+> only at the similarity step, by which point the patch features have already been computed and cached —
+> *"the failure surfaces far from its cause"* (`core/config.py:170-196`).
+
+---
+
+# Part IV — Contract-first development
+
+## 11. `core/schemas.py` is the binding contract
+
+`core/schemas.py` (462 lines) holds every model the system exchanges. The rule is simple: **no specialist
+may invent its own result shape.** Every specialist returns exactly `SpecialistResult`
+(`core/schemas.py:325-406`), whose docstring says so: *"Every specialist returns exactly this. No
+exceptions."*
+
+The models, in order:
+
+| Model | Line | Role |
+|---|---|---|
+| `Task` (enum) | 35 | the six-task ontology (`vqa`, `caption`, `grounding`, `change`, `optical_sar`, `change_vqa`) + `unsupported` |
+| `Modality` (enum) | 50 | `optical` / `sar` / `joint` |
+| `CoordinateSystem` (enum) | 57 | `normalized_0_1` / `geo` |
+| `EvidenceType` (enum) | 65 | the evidence classes |
+| `ControllerState` (enum) | 79 | the nine-state spine |
+| `Intent` | 94 | the router's output |
+| `GeoMetadata` | 120 | CRS / transform metadata |
+| `SensorDescriptor` | 138 | sensor identity |
+| `AssetMetadata` | 153 | one input asset |
+| `Box` | 171 | a flat box (see the flat-vs-nested trap below) |
+| `Region` / `ChangeRegion` | 189 / 202 | spatial outputs |
+| `Evidence` | 216 | one observable artefact |
+| `ConfidenceBreakdown` | 259 | measurable confidence |
+| `TraceStep` / `ModelRef` / `ExecutionTrace` | 279 / 288 / 296 | the trace |
+| `SpecialistResult` | 325 | the master contract |
+| `AnalysisRequest` | 412 | the request |
+| `ResultEnvelope` | 421 | the response wrapper |
+| `HealthStatus` | 430 | the health shape |
+
+**Every model sets `model_config = ConfigDict(extra="forbid")`.** This is deliberate and has a documented
+consequence (C-2, §5): the models reject a body carrying an unknown key, which contradicts
+`API_CONTRACT.md` §1.1's forward-compatibility promise. The code is authoritative; the document is the
+inaccurate half (`docs/STEP7_BACKEND_CHAIN_REPORT.md` §15).
+
+> **The flat-vs-nested trap.** `Box` is **flat**, not nested. An earlier documentation draft described it
+> with nested geometry, and *"every box the frontend drew would have been at the origin"*
+> (`docs/PHASE19_FINAL_HARDENING.md` §3.6). Documentation is validated against the real models by tests
+> (`test_api_contract_doc.py`, `test_frontend_guide_doc.py`), which is what caught it.
+
+### 11.1 The two cross-field validators you must not break
+
+`SpecialistResult` carries a `_task_output_consistency` validator (`core/schemas.py:353-406`) with two
+rules:
+
+- **Grounding with no localisation is degraded, not a crash.** If `task == GROUNDING` and there are no
+  boxes or regions, the validator appends a warning and sets `degraded = True`.
+- **Change-VQA with no answer text is degraded.** If `task == CHANGE_VQA` and the answer is blank, the
+  validator marks it degraded — but only if the specialist has not already set `degraded` itself.
+
+The `CHANGE` clause that once lived there was **removed** (F-16c): it had collapsed to `not regions`, and
+a *successful* no-change analysis began reporting `degraded: true`. **CHANGE is the one task whose
+`degraded` flag is now set entirely by its specialist** (`core/schemas.py:362-395`).
+
+## 12. The specialist interface
+
+Every specialist implements exactly one abstract base class, `Specialist`, in `specialists/base.py`. The
+interface is **four methods**, and the split is deliberate (`specialists/base.py:1-17`):
+
+```
+validate_request    -> can this specialist serve this request at all?
+execute             -> do the work, return a SpecialistResult
+produce_evidence    -> what observable artefacts support the result?
+estimate_confidence -> what measurable signals support the score?
+```
+
+The class attributes a specialist must set (`specialists/base.py:48-58`):
+
+| Attribute | Meaning |
+|---|---|
+| `name` | stable machine-readable name, used in traces and evidence sources |
+| `version` | semantic version; *"bump when behaviour changes, not when code moves"* |
+| `capabilities` | the capability strings this specialist serves, e.g. `("vqa", "caption")` |
+
+The four abstract methods (`specialists/base.py:62-96`):
+
+| Method | Contract |
+|---|---|
+| `validate_request(request)` | raise the **most specific** typed error available (`InvalidRequestError`, `PairMisalignmentError`, …), never a bare `Exception` |
+| `execute(request)` | return a normalised `SpecialistResult`; *"Never returns None."* |
+| `produce_evidence(result)` | return `list[Evidence]`; *"Must not invent anything the specialist did not actually compute."* |
+| `estimate_confidence(result)` | return a `ConfidenceBreakdown`; *"Never an LLM utterance."* |
+
+The base class also provides helpers a specialist should use rather than re-implement
+(`specialists/base.py:98-134`): `supports()`, `require_assets()` (asserts an exact asset count and raises
+a typed error), `require_capability()`, `model_refs()` (for the trace), and `describe()`.
+
+`SpecialistRequest` (`specialists/base.py:34-45`) is the input: `assets`, `query`, `params`, `run_id`.
+Its docstring states the isolation rule: *"Everything a specialist is given. No specialist reads global
+state."*
+
+## 13. How to add a new specialist
+
+This is the verified procedure, read from the code. There are **five** steps, and skipping any one fails
+loudly (which is the design).
+
+### Step 1 — implement the `Specialist` ABC
+
+Subclass `specialists.base.Specialist`, set `name`, `version` and `capabilities`, and implement the four
+methods. The builder for the class is a module-level function
+`build_<x>_specialist(config, *, device, **kwargs)` — the same shape as the four existing builders
+(`core/registry.py:185-259` names them: `build_vqa_specialist`, `build_grounding_specialist`,
+`build_change_specialist`, `build_change_vqa_specialist`, `build_optical_sar_specialist`).
+
+### Step 2 — add a row to the spec table
+
+Add a `SpecialistSpec` to `default_specs()` in `core/registry.py:171-260`. The spec's fields
+(`core/registry.py:121-165`):
+
+| Field | Meaning |
+|---|---|
+| `name` | the registry key — **the capability string**, never the specialist's own `name` |
+| `capabilities` | the tuple the built object must declare; **asserted** after construction |
+| `module` | dotted module path containing the builder |
+| `builder` | builder function name inside `module` |
+| `requires_assets` | exact asset count, or `None` for "any" |
+| `config_keys` | `{builder_kwarg: dotted.config.key}` — only keys that resolve are passed |
+| `optional_config_keys` | as above, but a missing key contributes nothing |
+| `failure_states` | registry state to use when construction raises, keyed by exception class name |
+
+The `SpecialistSpec.__post_init__` refuses a spec whose `name` is not in its own `capabilities`
+(`core/registry.py:158-165`): *"the registry keys on capability, so this spec would be unreachable."*
+
+### Step 3 — understand the capability-vs-name asymmetry
+
+**This is the trap the registry exists to encode** (`core/registry.py:30-45`). The VQA specialist declares:
+
+```python
+name = "vlm"                       # specialists/vqa/inference.py:114
+capabilities = ("vqa", "caption")  # specialists/vqa/inference.py:116
+```
+
+Every other specialist's `name` equals its single capability. A registry keyed on `name` would make `vqa`
+permanently unfindable while every other specialist kept working. So the registry keys on **capability**
+and **asserts** the capability tuple against the constructed object
+(`core/registry.py:497-515`). If your spec's `capabilities` disagrees with what your specialist declares,
+construction fails with a `SpecialistError` naming both tuples.
+
+### Step 4 — register the task (only if it is a *new* task)
+
+If the new specialist serves an existing task, nothing else is needed. If it is a new task, add it to:
+
+- `Task` in `core/schemas.py:35-49`, and
+- `TASK_CAPABILITY` in `core/planner.py:121-128`, and
+- `CAPABILITY_ASSETS` in `core/planner.py:133-142`.
+
+`CAPABILITY_ASSETS` mirrors each specialist's own `validate_request`, which **stays authoritative** — the
+planner's copy is a *"cheap precondition so it can refuse before construction is attempted"*
+(`core/planner.py:130-132`). A test asserts `CAPABILITY_ASSETS` equals `SpecialistSpec.requires_assets`
+in both directions, and asserts the gateway carries **no third copy** of the asset-count table
+(`docs/STEP7_BACKEND_CHAIN_REPORT.md` §3).
+
+> **Adding a task touches `router.tasks`.** `router.num_tasks` must equal `len(router.tasks)`
+> (`core/config.py:199-206`), so a new task means a config edit — **which moves the hash** (§8.2). This is
+> the one part of "add a specialist" that has a global consequence.
+
+### Step 5 — return the right shapes
+
+`execute` must return a `SpecialistResult`; `produce_evidence` must return `list[Evidence]`;
+`estimate_confidence` must return a `ConfidenceBreakdown`. The registry will construct your specialist
+lazily and record its state.
+
+### What the registry does with your specialist
+
+| Behaviour | Mechanism |
+|---|---|
+| lazy construction | the builder is imported via `importlib.import_module` **inside** `build()` — no module-level specialist import (`core/registry.py:15-25,462-475`) |
+| memoisation | the second `build(cap)` returns the cached entry (`core/registry.py:432-434`) |
+| three states | `AVAILABLE` / `DEGRADED` / `UNAVAILABLE` (`core/registry.py:102-107`) |
+| degradation detection | duck-typed on `has_checkpoint` / `has_head` / `has_encoder` / `model is None` (`core/registry.py:527-548`) |
+| failure is retained | a construction failure returns an `UNAVAILABLE` entry rather than raising; **only an unknown capability raises** (`core/registry.py:418-450`) |
+| corrupt ≠ missing | `ModelLoadError` / `ModelUnavailableError` map to `UNAVAILABLE`, never retried as `DEGRADED` (`core/registry.py:550-612`) |
+| path scrubbing | the client-visible `detail` is `scrub_paths(...)`; the raw string goes to the log only (`core/registry.py:560-598`) |
+
+> **The corrupt-vs-missing rule is load-bearing.** *"silently running an untrained model because a real
+> checkpoint failed to load would be the worst outcome"* (`specialists/change/specialist.py:834-838`,
+> quoted in `core/registry.py:63-75`). Do not add a fallback that re-adds a degradation the builder
+> refused.
+
+## 14. How evidence is emitted
+
+Evidence is produced by `Specialist.produce_evidence(result)` and returned as `list[Evidence]`. The
+`Evidence` model (`core/schemas.py:216-256`):
+
+| Field | Type | Note |
+|---|---|---|
+| `evidence_id` | `str` | auto-generated `ev_<hex>`; **must be unique within a result** |
+| `type` | `EvidenceType` | the evidence class |
+| `source_specialist` | `str` | which specialist computed it |
+| `coordinate_system` | `CoordinateSystem \| None` | required for spatial evidence |
+| `coordinates` | `list[float] \| None` | the geometry |
+| `score` | `float \| None` | `0.0 ≤ score ≤ 1.0` |
+| `artifact_ref` | `str \| None` | **never a filesystem path** (F-16) |
+| `payload` | `dict[str, Any]` | structured detail |
+
+Two validators enforce correctness:
+
+- `SpecialistResult._unique_evidence_ids` rejects a result with duplicate `evidence_id` values
+  (`core/schemas.py:345-351`).
+- `Evidence._spatial_needs_crs` rejects spatial evidence (`BOUNDING_BOX`, `MASK`, `CHANGE_MAP`, `TILE`,
+  `IMAGE_CROP`, `JOINT_FEATURE_REGION`) that carries coordinates but **no** `coordinate_system`
+  (`core/schemas.py:241-255`).
+
+> **`artifact_ref` is never a filesystem path.** v1 exposes no artifact-serving endpoint, so it is null
+> unless a deployment supplies a client-fetchable reference. *"An artifact may still be written
+> server-side where configured; being written is not the same as being retrievable."*
+> (`core/schemas.py:227-238`.)
+
+---
+
+# Part V — The test workflow
+
+## 15. Running tests
+
+### 15.1 Always use the repository venv interpreter
+
+pytest is installed **only** in the repository virtualenv. Invoking the system `pytest` fails or resolves
+to a different interpreter (`release/repo/docs/REPRODUCIBILITY.md` §10.2):
+
+```bash
+.venv/Scripts/python.exe -m pytest tests/unit/test_frontend_live_wiring.py -q
+```
+
+On Windows, pytest must be given `-p no:cacheprovider` because the sandbox refuses `.pytest_cache` writes
+(`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §0).
+
+### 15.2 Run targeted files, not the whole tree
+
+**A full `tests/unit` run trips the sandbox's bulk-delete guard** (4× `test_safe_delete_shim` failures)
+(`release/repo/docs/REPRODUCIBILITY.md` §10.3). The workaround is to run the targeted suites:
+
+```bash
+.venv/Scripts/python.exe -m pytest tests/unit/test_frontend_live_wiring.py -q     # 106 passed
+```
+
+The runbook notes that **multi-suite invocations in one command have been refused by the environment
+before; single suites are reliable** (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §2.6):
+
+```bash
+$PY -m pytest tests/unit/test_deploy_config.py -p no:cacheprovider -q
+$PY -m pytest tests/unit/test_app_serving.py -p no:cacheprovider -q
+$PY -m pytest tests/unit/test_api_contract_doc.py tests/unit/test_frontend_guide_doc.py -p no:cacheprovider -q
+```
+
+> **Do not pipe pytest through `grep`** in the authoring sandbox — output is block-buffered and a killed
+> pipeline swallows it. Redirect to a file instead (`release/repo/docs/REPRODUCIBILITY.md` §5.6).
+
+### 15.3 `pytest.ini`
+
+```ini
+[pytest]
+testpaths = tests
+pythonpath = .
+addopts = -q --tb=short
+```
+
+(`pytest.ini:1-4`)
+
+`pythonpath = .` is what makes `from core.config import ...` resolve from the repo root without an
+installed package.
+
+## 16. The evidence-class markers
+
+`pytest.ini` registers four markers whose purpose is to make a test's **evidence class** explicit
+(`pytest.ini:23-27`):
+
+| Marker | Meaning |
+|---|---|
+| `unit` | fast, no I/O, no server, no network. Evidence about a component in isolation. |
+| `integration` | exercises two or more real components wired together in-process. |
+| `smoke` | a minimal end-to-end path run against real local artifacts, not a mock. |
+| `real_inference` | ran the real model on real inputs in this environment. |
+
+The file's comment states the discipline:
+
+> *"a unit test never claims a deployment was exercised, and no test may be reported as 'real inference'
+> unless it ran the real model on real data in this environment."* (`pytest.ini:13-17`)
+
+**`environment_blocked` is deliberately not a marker.** *"a blocked path is reported in the STEP 7 report
+rather than encoded as a permanently-skipped test, because a skip can be mistaken for coverage"*
+(`pytest.ini:19-21`). An unregistered mark is an **error** rather than a silent no-op (`pytest.ini:14-15`).
+
+## 17. The known failures, and why they are not regressions
+
+Running the entire unit tree trips **5–6** failures, classified by cause
+(`release/repo/docs/REPRODUCIBILITY.md` §5.4):
+
+| # | Failure | Cause | Regression? |
+|---|---|---|---|
+| 1–4 | `test_safe_delete_shim` (×4) | the sandbox's bulk-**delete guard** (Windows verbatim-path behaviour) | **No** |
+| 5 | one ordering flake in the router route test | passes in isolation; order/collection-dependent | **No** |
+| 6 | one stale adapter test | asserts `optical_sar` absent when CROMA is *unshipped* — **CROMA is now shipped** | **No** |
+
+Re-running the affected files together passes **137** tests, which is what isolates them as environmental
+rather than behavioural (`release/repo/docs/REPRODUCIBILITY.md` §5.4).
+
+**Known per-suite results:**
+
+| Suite | Command | Expected |
+|---|---|---|
+| Frontend live-wiring | `pytest tests/unit/test_frontend_live_wiring.py` | **106 passed** |
+| Doc/frontend suite | `pytest` on the 5 doc/frontend files | **183 passed** |
+| Full unit suite | `pytest tests/unit` | 5–6 **environmental** failures |
+| Gateway policy | `pytest tests/unit/test_gateway_policy.py` | **51 passed** |
+| Evidence engine | `pytest tests/unit/test_evidence_engine.py` | **73 passed** |
+
+(`release/repo/README.md` §The test suites; `docs/PHASE19_FINAL_HARDENING.md` §6.)
+
+> **The precise full-suite *collected* count is `UNKNOWN — not established from the available evidence`.**
+> Per-suite counts are known; the single collected total is not (`release/repo/docs/REPRODUCIBILITY.md`
+> §5.4).
+
+---
+
+# Part VI — Scripts and notebooks
+
+## 18. The class of scripts under `scripts/`
+
+`scripts/` is a **flat** directory of executable helpers — no sub-packages. The listing holds **51 script
+files** (47 `.py`, 3 `.ps1`, 1 `.mjs`) plus `__init__.py`. They fall into clear classes:
+
+| Class | Examples | Purpose |
+|---|---|---|
+| **Training entry points** | `train_router.py`, `train_grounding.py`, `train_change.py`, `train_fusion.py`, `train_change_vqa.py` | thin CLIs over the `training/` modules (§20) |
+| **Evaluation** | `eval_change.py`, `eval_fusion_115.py`, `eval_grounding_head.py`, `evaluate_change_vqa.py` | per-task evaluation |
+| **Data preparation** | `prepare_bigearthnet.py`, `prepare_change_vqa.py`, `select_bigearthnet_slice.py` | build corpora and selection manifests |
+| **Contract probes** | `probe_grounding_head_contract.py`, `probe_remoteclip_contract.py`, `probe_vlm_contract.py` | measure a real model's contract before relying on it |
+| **Smoke tests** | `smoke_test_change.py`, `smoke_test_grounding_head.py`, `smoke_test_vlm.py` | a minimal real path per specialist |
+| **Verification / gates** | `verify_gate1.py`, `verify_croma_forward.py`, `verify_grounding_e2e.py`, `verify_levir_real.py`, `verify_cdvqa_imagery.py` | prove a property end to end |
+| **Threshold / hyperparameter sweeps** | `sweep_change_threshold.py`, `sweep_router_threshold.py`, `fusion_seed_variance.py` | sweep a frozen knob |
+| **Phase-6 (VLM) tooling** | `phase6_train_vlm.py`, `phase6_adjudicate_test.py`, `phase6_close.py`, `phase6_rerule.py`, `phase6_recover_baseline_test.py` | the VLM acceptance workflow |
+| **Phase-12 (fusion) tooling** | `p12_integrity_verify.py`, `p12_preflight_verify.py`, `run_phase12_extraction.ps1`, `run_phase12_resume_ref.ps1`, `run_phase12_resume_s1_ref.ps1` | the fusion feature-extraction workflow (PowerShell drivers) |
+| **Calibration** | `fit_calibration.py` | fit the temperature-scaling artifact on validation (§22) |
+| **Kaggle packaging / rehearsal** | `package_kaggle_code.py`, `rehearse_kaggle_notebook.py`, `rehearse_change_notebook.py` | package and dry-run the notebooks |
+| **Deployment validation** | `validate_deploy_config.py` | prove `configs/deploy.yaml` is inert |
+| **Frontend staging** | `stage_pages.mjs` | build the Cloudflare Pages bundle |
+| **Environment / diagnostics** | `check_env.py`, `diagnose_feature_cache.py`, `croma_normalisation_arm_probe.py`, `analyze_grounding_resolution.py`, `analyze_reben_labels.py`, `check_cdvqa_second_overlap.py`, `establish_cdvqa_temporal_order.py`, `exp_grounding_resolution.py`, `extract_fusion_features.py` | environment and dataset diagnostics |
+
+**The training entry points share one convention**, stated in their docstrings: *"A THIN CLI over
+`training/<task>/train.py` … Everything that computes lives in the module; this file parses arguments,
+reports the environment, prints the accounting, and returns an exit code."* (`scripts/train_change.py:1-6`,
+`scripts/train_fusion.py:1-6`). The exit codes are a contract (`scripts/train_change.py:17-24`):
+
+```
+0   training completed (or --dry-run validated a present dataset)
+2   the dataset is missing, empty, or cannot be split -- NOT a crash
+3   the run started and the trainer raised a typed error
+```
+
+> **Re-exported names are part of the contract.** Several scripts re-export their trainer's symbols at
+> module scope, and a test asserts it. E.g. `tests/unit/test_change_train_script_contract.py` asserts
+> `change_loss`, `train_change_head` and `evaluate` are reachable as `scripts.train_change.*`
+> (`scripts/train_change.py:26-30`).
+
+## 19. `notebooks/`
+
+| Notebook | Purpose |
+|---|---|
+| `kaggle_change_vqa_train.ipynb` | R-02 change-VQA training (§21) |
+| `kaggle_phase6_vlm_lora.ipynb` | Phase-6 SmolVLM LoRA training (§21) |
+| `kaggle_change_training.ipynb` | the change **detector** — *"frozen and out of scope"* (`docs/R02_KAGGLE_TRAINING_GUIDE.md` §4) |
+| `kaggle_grounding_resolution.ipynb` | Phase-7 grounding resolution — unrelated to R-02 |
+
+> **Do not confuse the two change notebooks.** For change-VQA use `kaggle_change_vqa_train.ipynb`; do
+> **not** use `kaggle_change_training.ipynb` (that one trains the change *detector*) or
+> `kaggle_grounding_resolution.ipynb` (Phase 7) (`docs/R02_KAGGLE_TRAINING_GUIDE.md` §4).
+
+Two `.bak-*` files sit beside the Phase-6 notebook; they are editor backups, not runnable notebooks.
+
+---
+
+# Part VII — Training entry points
+
+The six artifacts train in two places: four locally, two on an external GPU. This is a **deliberate
+boundary** — the release ships *"frozen artifacts with provenance, not a retraining harness"*
+(`release/repo/docs/REPRODUCIBILITY.md` §8.5).
+
+| Artifact | Where it trains | Reproducible from this release? |
+|---|---|---|
+| router adapter | local CPU | **yes** — `configs/base.yaml` §`router.training` |
+| grounding head | local | **yes** — `configs/base.yaml` §`grounding_training` |
+| change head | local | **yes** — `configs/base.yaml` §`change` |
+| optical_sar fusion head | local, seed sweep | **yes** |
+| change_vqa head | **external GPU (Kaggle)** | **partly** — the promotion gate, evaluation and serving wiring are reproducible; there is no one-command retrain |
+| vlm LoRA adapter | **external GPU** | **partly** — same |
+
+(`release/repo/README.md` §What "reproduce" means.)
+
+## 20. Local training (router, grounding, change, optical-SAR)
+
+### 20.1 Router — CPU-only, and fast
+
+```bash
+python scripts/train_router.py --smoke           # fast, stub encoder
+python scripts/train_router.py --stub            # fast, no model download
+python scripts/train_router.py                   # full run, real MiniLM
+```
+
+(`scripts/train_router.py:1-10`)
+
+The script's own measured note: *"Runs entirely on CPU. Measured cost with the real encoder: ~40 s to load
+MiniLM on a cold cache, ~0.1 s to embed the corpus, ~1 s to train 60 epochs. There is no reason to spend
+Kaggle GPU quota on this."* The encoder is frozen, so embeddings are cached and the adapter trains on
+cached vectors (`configs/base.yaml:67-69`).
+
+### 20.2 Grounding — two separable stages
+
+```bash
+# stage 1 only — measure the corpus before committing to training
+python scripts/train_grounding.py --data-root <root> --extract-only
+
+# both stages, real run
+python scripts/train_grounding.py --data-root <root> --checkpoint <ckpt.pt>
+
+# 2-batch forward+backward+checkpoint+reload, on CPU
+python scripts/train_grounding.py --data-root <root> --debug
+```
+
+(`scripts/train_grounding.py:6-22`)
+
+Splitting extraction from training means a hyperparameter change does not re-encode 15,699 images, and a
+crashed training run does not lose the cache. The stated bar: *"the Phase 7 zero-shot baseline scored
+0.0972 mean best IoU over 16,159 real eval records. This head must beat it by MIN_IMPROVEMENT_IOU to
+justify existing. The comparison is printed whether or not it passes."*
+
+### 20.3 Change — a thin CLI over `training/change/train.py`
+
+```bash
+# validate the data and the split; build nothing, train nothing
+python scripts/train_change.py --data-root <LEVIR-CD root> --dry-run
+
+# real run on CPU
+python scripts/train_change.py --data-root <LEVIR-CD root> --device cpu
+
+# a first real-data run that does not commit an hour
+python scripts/train_change.py --data-root <root> --limit 256 --epochs 3
+```
+
+(`scripts/train_change.py:8-16`)
+
+`--dry-run` is *"the leakage check without the cost: it loads the items, performs the scene-disjoint
+split, runs `assert_image_disjoint`, prints the accounting and exits. It does NOT build the detector, so
+it cannot trigger a pretrained-weights download, and it writes no files."*
+
+### 20.4 Optical-SAR fusion — CPU-only, no result claimed
+
+```bash
+# validate the caches and the split; train nothing, write nothing
+python scripts/train_fusion.py --train-cache train.npz --val-cache val.npz --dry-run
+
+# a real run on CPU (the fusion head is CPU-only)
+python scripts/train_fusion.py --train-cache train.npz --val-cache val.npz --arm A --epochs 20
+```
+
+(`scripts/train_fusion.py:8-15`)
+
+> **No performance number is a result here.** The script states it: *"This loop has never seen the real
+> paired BigEarthNet-S1+S2 corpus. Every run record it writes carries `result_status` and
+> `pre_registered_metric_computed: false`; the pre-registered 11.5 metric is not computed."*
+> (`scripts/train_fusion.py:17-20`.)
+
+## 21. External GPU training (change-VQA, VLM LoRA)
+
+Both external runs happen on **Kaggle** with **GPU T4 ×2**. Neither is a one-command retrain from this
+release.
+
+### 21.1 Change-VQA (R-02)
+
+**Guide:** `docs/R02_KAGGLE_TRAINING_GUIDE.md` (37 KB). **Runbook:** `RUNBOOK_CHANGE_VQA_KAGGLE.md`.
+
+| Property | Value | Source |
+|---|---|---|
+| notebook | `notebooks/kaggle_change_vqa_train.ipynb` | `docs/R02_KAGGLE_TRAINING_GUIDE.md` §4 |
+| accelerator | **GPU T4 ×2** | §5 |
+| internet | **On** (MiniLM downloads) | §5 |
+| cells | **all 34, top to bottom** | §7 |
+| hard stop | `HARD_STOP_SECONDS = 3 * 3600` (the plan's 3 h) | §7 |
+| trainer defaults | `epochs=40, batch_size=128, seed=42, patience=6, time_limit=10800s` | §7 |
+
+The path-discovery cells find the code root (four markers) and the CDVQA root (12 annotations + 4 image
+dirs), and section 3c verifies the frozen STANet checkpoint by size and SHA256
+(`RUNBOOK_CHANGE_VQA_KAGGLE.md` §5). **Test splits are not readable by the trainer:**
+`training/change_vqa/train.py` loads only Train and Val, and *"there is no option in either file that
+changes that"* (`scripts/train_change_vqa.py:11-15`).
+
+### 21.2 VLM LoRA (Phase 6)
+
+**Runbook:** `RUNBOOK_PHASE6_VLM_KAGGLE.md`.
+
+| Property | Value | Source |
+|---|---|---|
+| notebook | `notebooks/kaggle_phase6_vlm_lora.ipynb` | `RUNBOOK_PHASE6_VLM_KAGGLE.md` §1 |
+| accelerator | **GPU T4 ×2** | §4 |
+| internet | **off** (base model attached as input) | §4 |
+| precision | **`fp16`, not the plan's `bf16`** — T4 is SM 7.5 | §4 |
+| base model | `HuggingFaceTB/SmolVLM-500M-Instruct`, ~1.02 GB `model.safetensors` | §4 |
+
+The trainer raises a `ConfigError` on `bf16` rather than silently falling back, and the deviation is
+recorded in the manifest under `plan_deviations` (`RUNBOOK_PHASE6_VLM_KAGGLE.md` §4).
+
+> **The VLM adapter's status is `ACCEPTANCE-REJECTED`.** Its metrics are *usable* (`exact_match 0.963`)
+> but it was not promoted. **USABLE ≠ ACCEPTED** (`DOCS_STYLE_GUIDE.md` §3). Do not describe the VLM path
+> as accepted.
+
+## 22. Calibration
+
+```bash
+python scripts/fit_calibration.py --dry-run    # check inputs, exit
+python scripts/fit_calibration.py              # fit, write artifact
+```
+
+(`scripts/fit_calibration.py:19-21`)
+
+It fits on **validation** data, and both the fitter entry point and the artifact writer **refuse a
+held-out split** — *"Fitting on Test or Test2 would make the reported confidence a function of the answers
+it is used to score — a leak, not a calibration."* (`scripts/fit_calibration.py:7-13`.)
+
+> **Calibration made ECE worse** — `0.013755 → 0.014929` — and is **retained only because it is in the
+> frozen config**. Never present it as an improvement (`DOCS_STYLE_GUIDE.md` §3).
+
+---
+
+# Part VIII — Coding conventions
+
+## 23. What the code actually does
+
+These conventions are visible across the files read. They are not aspirational style rules; each is
+observable in the code.
+
+### 23.1 Every module has a substantial docstring stating *why*
+
+The files read are densely commented at the module and function level, and the comments explain decisions
+and failure modes rather than restating the code. Examples: `core/registry.py:1-76` (a 76-line module
+docstring on the capability-key trap), `deploy/codespace/launch.sh:1-24` (why the launcher is defensive),
+`gateway/app.py:53-106` (the annotation-scope trap). **A change that removes the reasoning from a
+comment removes the reason a future reader will not re-introduce the bug.**
+
+### 23.2 `from __future__ import annotations` is standard
+
+It appears at the top of nearly every module (`core/config.py:11`, `core/registry.py:78`,
+`specialists/base.py:19`, `gateway/app.py:48`, `deploy/render/main.py:40`, `deploy/render/codespaces.py:16`,
+`deploy/codespace/warm_cache.py:28`, `scripts/*.py`). It is convenient — and it is the direct cause of
+Trap 6 (§30).
+
+### 23.3 Findings are recorded as short codes, inline
+
+The code refers to findings by code (`C-1`, `C-6`, `C-8`, `F5-2`, `P7-1`, `F-15`, `F-16`, `F-17`) and
+states the failure each guard prevents. This is a documentation convention enforced by comments, e.g.
+`core/config.py:121-142` (finding F5-2, the 17× overrun), `core/registry.py:227-235` (F-17, a config
+surface removed rather than left as a silent no-op).
+
+### 23.4 Pydantic models set `extra="forbid"` and carry validators
+
+Every schema model forbids extra fields and uses `@field_validator` / `@model_validator` for cross-field
+rules (§11). New models should follow the same pattern.
+
+### 23.5 Loggers are module-level and named `satquery.<area>`
+
+`logging.getLogger("satquery.orchestrator")` (`deploy/render/main.py:74`),
+`logging.getLogger("satquery.registry")` (`core/registry.py:95`), `logging.getLogger(__name__)`
+(`gateway/app.py:123`). Log calls use `%s`-style free text — there are no structured/JSON logs
+(`docs/architecture/10-observability-and-ops.md` §4.4).
+
+### 23.6 Client-visible strings are scrubbed
+
+Server-side diagnostics that name absolute paths must not reach a client. `core/errors.scrub_paths` reduces
+a path to a basename before it is published (`core/registry.py:297-312,560-598`;
+`core/errors.py:39`). Any new client-visible field that could carry a path or an exception string must go
+through the same scrub.
+
+### 23.7 `noqa` comments state the reason
+
+Where a lint suppression is used, the reason is written, e.g. `# noqa: E402  (see comment above)` in
+`gateway/app.py:85-106`, and `# noqa: BLE001 - report, never crash the warm step` in
+`deploy/codespace/warm_cache.py:74`.
+
+### 23.8 Tests assert the *cause*, not the symptom
+
+The regression test for the annotation-scope trap asserts that the route has no query parameters and that
+`gateway.app.Request is starlette.requests.Request` — *"rather than the symptom, because asserting the
+symptom would be brittle"* (`docs/STEP7_BACKEND_CHAIN_REPORT.md` §13). Follow this when writing a
+regression test.
+
+### 23.9 The 88-column soft limit
+
+The files read wrap around 88 characters. There is no committed linter config in the files read, so this
+is a convention, not an enforced rule — **the exact formatter/linter configuration is `UNKNOWN — not
+established from the available evidence`.**
+
+---
+
+# Part IX — Known development traps
+
+## 24. The traps, in one table
+
+| # | Trap | One-line consequence |
+|---|---|---|
+| 1 | the monorepo `deploy/` is stale and untracked | it is **not** the deployed source |
+| 2 | the sandbox proxy is dead | outbound calls need `--noproxy '*'` / `ProxyHandler({})` |
+| 3 | pytest exists only in the venv | the system `pytest` resolves to the wrong interpreter |
+| 4 | Chrome drops synthetic CDP key events without OS focus | a browser-driven run silently answers the default query |
+| 5 | Cloudflare `_headers` rules concatenate | a later rule cannot "fix" an earlier one |
+| 6 | `from __future__ import annotations` + FastAPI | an unresolvable `Request` annotation becomes a **required query parameter** |
+| 7 | the full-suite run trips the bulk-delete guard | 4 spurious `test_safe_delete_shim` failures |
+| 8 | a stale serve process keeps answering | it reports the **previous revision's** capabilities |
+
+## 25. Trap 1 — the stale untracked `deploy/`
+
+**Symptom.** You edit `deploy/render/main.py`, deploy your change, and nothing changes in production — or
+you read `deploy/render/main.py` and cannot find the tunnel code the live service runs.
+
+**Root cause.** `deploy/` inside the monorepo is **stale and untracked**. `git status` reports
+`?? deploy/` (verified in the working copy). It is **not** the deployed source
+(`release/repo/docs/DEPLOYMENT.md` §1; `release/CURRENT_RELEASE_STATE.md` §6).
+
+**Evidence of divergence.** The monorepo `deploy/render/main.py` (532 lines) exposes `/api/health` with a
+`config` block that has **no** `tunnel` field and no `transport_mode` / `tunnel_timeout_s` /
+`wake_timeout_s` keys (`deploy/render/main.py:444-466`), whereas the **live** payload carries all of them
+(`release/repo/docs/DEPLOYMENT.md` §5). The monorepo copy also lacks `tunnel_agent.py` and `doctor.sh`,
+both of which `launch.sh` references (`deploy/codespace/launch.sh:83,89,153,160-168,179`).
+
+**Fix.** Fetch the deployed file from the private repository and diff before editing. Treat the monorepo
+`deploy/` as documentation of intent, not as source.
+
+## 26. Trap 2 — the dead sandbox proxy needs `--noproxy '*'`
+
+**Symptom.** Outbound HTTP calls fail or hang in the authoring sandbox.
+
+**Root cause.** The sandbox proxy is dead; requests are routed to it and never reach the target.
+
+**Fix.** Disable proxies for the call (`release/repo/docs/REPRODUCIBILITY.md` §10.1):
+
+```bash
+curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/health
+```
+
+```python
+opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
+```
+
+`release/tools/hf_verify.py:37-38` does exactly this (`opener_no_proxy()`). In a normal environment the
+flag is harmless; in the sandbox it is mandatory.
+
+## 27. Trap 3 — pytest only in the venv
+
+**Symptom.** Invoking the system `pytest` fails or resolves to a different interpreter.
+
+**Root cause.** pytest is installed only in `.venv`.
+
+**Fix.** Always invoke the venv interpreter explicitly (`release/repo/docs/REPRODUCIBILITY.md` §10.2):
+
+```bash
+.venv/Scripts/python.exe -m pytest tests/unit/test_frontend_live_wiring.py -q
+```
+
+## 28. Trap 4 — Chrome drops synthetic CDP key events
+
+**Symptom.** A browser-driven run silently answers the *default* query; the query box looks untouched;
+`mock_nodes` is non-zero.
+
+**Root cause.** Chrome **drops synthesized key events when the browser window does not hold OS focus**.
+`press_key` / `fill_input` (real CDP key events) are focus-gated; `Input.insertText` (`type_text`) is not.
+
+**Measured.** With Chrome backgrounded, `press_key("Z")` left `#qtext.value` unchanged, while
+`type_text("Q")` inserted fine (`release/repo/docs/REPRODUCIBILITY.md` §6.4).
+
+**Fix.** Use `type_text` (not `fill_input`), and **assert the input state before dispatch** — `q_ok`
+(the query box really held the query), `obs_ok` (`#obsTail == 'ready'`), `t0_ok` (both frames present
+where required). This is the single most dangerous trap because it produces a **silent false pass** — the
+pipeline "works", it just answered a different question
+(`release/repo/docs/REPRODUCIBILITY.md` §10.5).
+
+## 29. Trap 5 — Cloudflare `_headers` concatenate
+
+**Symptom.** A specific cache-control rule does not take effect; the browser caches a file you expected it
+to revalidate.
+
+**Root cause.** Cloudflare `_headers` rules **concatenate, they do not override.** Two matching rules are
+merged: a specific rule nested under a broad `/assets/img/*` rule yields
+`max-age=604800, …, max-age=0, must-revalidate` — and **Chromium takes the FIRST `max-age`**. The file's
+own "later rules override" comment is **false** (`release/repo/docs/DEPLOYMENT.md` §10).
+
+**Fix.** Order rules so the *broadest* rule appears last, and never rely on a later rule overriding an
+earlier one. Related: Cloudflare **308-redirects `X.html` → `/X`**, so reference the extensionless path
+(`release/repo/docs/REPRODUCIBILITY.md` §10.4).
+
+## 30. Trap 6 — the annotation-scope trap
+
+**Symptom.** Every `POST` route returns `422` with FastAPI's own shape, without ever entering the handler:
+
+```
+POST /v1/analyze  →  422
+{"detail":[{"type":"missing","loc":["query","request"],"msg":"Field required"}]}
+```
+
+(`docs/STEP7_BACKEND_CHAIN_REPORT.md` §13.)
+
+**Root cause.** The module uses `from __future__ import annotations`, so every annotation is a **string**
+at runtime. FastAPI resolves those strings via `get_typed_signature`, which calls
+`eval(annotation, func.__globals__)`. If `Request` is imported **inside** `create_app`, the route closures
+capture the name as a *local* of `create_app`; it never appears in `gateway.app.__globals__`, so
+resolution fails and FastAPI is left holding a bare `ForwardRef('Request')`. **FastAPI does not raise** —
+it silently falls back to treating the parameter as a **required query parameter named `request`**
+(`gateway/app.py:53-73`).
+
+Three things were wrong at once: the request body was never read, the gateway's own validation never ran,
+and the error shape was FastAPI's `{"detail": ...}` rather than the contract's `{"error": {...}}`. **Every**
+POST route was affected, including `/v1/assets`.
+
+**The asymmetry to internalise.** An unresolvable **parameter** annotation is *silently reinterpreted*
+(the handler never runs), while an unresolvable **return** annotation *raises*
+(`pydantic.errors.PydanticUndefinedAnnotation: name 'JSONResponse' is not defined`, which made
+`create_app()` unbuildable). Same root cause, opposite diagnosability
+(`gateway/app.py:88-106`).
+
+**Fix.** Bind the annotation subjects at **module scope** (`gateway/app.py:79-106`):
+
+```python
+if TYPE_CHECKING:  # pragma: no cover
+    from starlette.requests import Request
+    from starlette.responses import Response
+
+#: Runtime bindings used as annotation subjects in this module. Deliberately
+#: module-level so `eval()` can find them. See the comment above.
+from starlette.requests import Request  # noqa: E402  (see comment above)
+from starlette.responses import Response  # noqa: E402  (see comment above)
+from fastapi.responses import JSONResponse  # noqa: E402  (see comment above)
+```
+
+The regression test asserts the **cause**: the route has no query parameters, and
+`gateway.app.Request is starlette.requests.Request`.
+
+> **The general rule.** Any name used as a FastAPI route annotation in a module with
+> `from __future__ import annotations` **must** be importable from that module's globals. Import it at
+> module scope, not inside the factory.
+
+## 31. Trap 7 — the full-suite bulk-delete guard
+
+**Symptom.** Running the whole `tests/unit` tree trips 4× `test_safe_delete_shim` failures.
+
+**Root cause.** Windows **verbatim-path** defects in the sandbox's bulk-delete guard. The precise
+condition under which the shim intermittently triggers on Windows is `UNKNOWN — not established from the
+available evidence` (`release/repo/docs/REPRODUCIBILITY.md` §10.3).
+
+**Fix / workaround.** Run the targeted suites (106 and 183 pass cleanly); treat the 4 shim failures as
+environmental, not regressions (§17).
+
+## 32. Trap 8 — the stale serve process
+
+**Symptom.** The Codespace answers `/v1/health` and `/v1/capabilities`, but reports the **previous
+revision's** capabilities.
+
+**Root cause.** A serve process started before a code or environment change keeps serving from old code.
+The serve process reads its environment exactly once, at startup (`deploy/codespace/launch.sh:100-102`).
+
+**Fix.** `launch.sh` already handles it: it records a **stamp** of the revision and the asset
+configuration and restarts the server when the stamp disagrees
+(`deploy/codespace/launch.sh:100-147`). The rule for a developer: **a restart, not a reload, is required
+after any env or revision change.**
+
+> *"A stale serve process is worse than no process: it answers /v1/health and /v1/capabilities from OLD
+> code, so the deployment looks alive while reporting the previous revision's capabilities."*
+> (`deploy/codespace/launch.sh:111-113`)
+
+### Related platform traps worth knowing
+
+| Trap | Detail |
+|---|---|
+| a forwarded Codespace port returns `302` for a private repo | this is *why* the outbound tunnel exists (`release/repo/docs/DEPLOYMENT.md` §10) |
+| the tunnel agent must be started by the devcontainer `postStartCommand` | a restarted Codespace comes up with `agent_connected: false` otherwise |
+| never retry `POST /api/infer` at the gateway | a retry consumes inference twice |
+| `containerEnv` applies only at container **creation** | an env change needs a restart, which is why `launch.sh` re-exports on every start (`deploy/codespace/launch.sh:49-52`) |
+| the Codespace filesystem is **ephemeral** | uploaded assets and logs vanish with the Codespace (`deploy/codespace/launch.sh:44-47`) |
+
+---
+
+# Part X — Status and evidence
+
+## 33. `NOT RUN` / `OPEN` / `BLOCKED` / `UNKNOWN` for development
+
+| # | Item | Status |
+|---|---|---|
+| 1 | B-07 — tunnel gaps; patch prepared, **not deployed** | **`OPEN`** |
+| 2 | B-02 — `/api/health` `codespace_name` trailing `\n` | **`OPEN` (cosmetic)** |
+| 3 | A `LICENSE` file | **`OPEN`** — **no LICENSE file exists**; README says to add one before public release |
+| 4 | The monorepo README | **materially stale** — it calls the frontend "hermetic", describes a 4-endpoint `/v1/*` contract, omits the tunnel, and points at the stale `deploy/` (`release/CURRENT_RELEASE_STATE.md` §6) |
+| 5 | `hf/SETUP.md` and `hf/README.md` | **stale** — they assert the project ships no weights, which is now false (`release/CURRENT_RELEASE_STATE.md` §6) |
+| 6 | The exact formatter / linter configuration | **`UNKNOWN`** — no committed config in the files read (§23.9) |
+| 7 | The precise full-suite collected test count | **`UNKNOWN`** — per-suite counts are known, the total is not |
+| 8 | The exact Windows trigger for the delete-shim flake | **`UNKNOWN`** |
+| 9 | Whether `doctor.sh` / `tunnel_agent.py` exist in the deployed inference repo | **`UNKNOWN`** — the monorepo copy lacks them |
+| 10 | A system-level end-to-end benchmark | **`NOT RUN`** — none exists |
+| 11 | The router **test**-split number | **`NOT RUN`** — only validation (`n = 86`, ungated) exists |
+| 12 | Captioning benchmark | **`NOT RUN`** — implemented, not benchmarked |
+| 13 | A one-command retrain for the two external artifacts | **not implemented** — deliberate boundary (§21) |
+| 14 | `torch.compile`, CUDA, quantisation, thread capping | **not implemented** — CPU-first by design (§1) |
+| 15 | Enforcement of a one-model cache (`cache_max_models: 1`) | **UNVERIFIED** — the value is reported, not proven enforced (`docs/DEPLOYMENT_DECISION.md` §5) |
+
+> **The stale-README trap is worth its own line.** `README.md` in the monorepo is *"materially stale"*: it
+> calls the frontend *"hermetic ��� no backend calls"* (it calls `/api/*` on Render), puts Render/Codespace
+> as *"in progress"* (both deployed), describes a 4-endpoint `/v1/*` contract (the live contract is
+> `/api/*`), omits the tunnel, and points at the stale untracked `deploy/` as the deployment source
+> (`release/CURRENT_RELEASE_STATE.md` §6). The public release documentation is authoritative; the
+> monorepo README is not.
+
+## 34. Where the evidence lives
+
+| What | Where |
+|---|---|
+| the frozen config, invariants and hash | `configs/base.yaml`; `core/config.py:76-80,94-222` |
+| the binding schemas | `core/schemas.py` |
+| the specialist interface | `specialists/base.py` |
+| the capability table and lazy construction | `core/registry.py` |
+| the planner mapping | `core/planner.py:121-142` |
+| the dependency profiles and contract notes | `requirements.txt` |
+| the test markers | `pytest.ini` |
+| the launcher and warm-up | `deploy/codespace/launch.sh`, `deploy/codespace/warm_cache.py` |
+| the annotation-scope trap | `gateway/app.py:53-106`; `docs/STEP7_BACKEND_CHAIN_REPORT.md` §13 |
+| the deploy manifest is inert | `docs/PHASE18_DEPLOYMENT_PACKAGING.md`; `scripts/validate_deploy_config.py` |
+| the change-VQA Kaggle guide | `docs/R02_KAGGLE_TRAINING_GUIDE.md`; `RUNBOOK_CHANGE_VQA_KAGGLE.md` |
+| the VLM LoRA Kaggle runbook | `RUNBOOK_PHASE6_VLM_KAGGLE.md` |
+| the test suites and their expected results | `release/repo/docs/REPRODUCIBILITY.md` §5 |
+| the environment traps | `release/repo/docs/REPRODUCIBILITY.md` §10 |
+| the platform traps | `release/repo/docs/DEPLOYMENT.md` §10 |
+| the live topology, env vars, cold start | `release/repo/docs/DEPLOYMENT.md` |
+| the operations manual | [../OPERATIONS.md](OPERATIONS.md) |
+| the factual inventory | `release/CURRENT_RELEASE_STATE.md` |
+
+### Cross-references
+
+| For | See |
+|---|---|
+| the frozen config and `Config.hash == 78f1e3700da15aa1` | [07 — Configuration and Freeze](architecture/07-configuration-freeze.md) |
+| the specialist contract in depth | [05 — Specialists](architecture/05-specialists.md) |
+| the router's five heads and the label space | [04 — Router](architecture/04-router.md) |
+| the evidence and confidence stages | [06 — Evidence and Confidence](architecture/06-evidence-and-confidence.md) |
+| the request lifecycle and the nine-state spine | [03 — Request Lifecycle](architecture/03-request-lifecycle.md) |
+| the four endpoints and error codes | [08 — The API Contract](architecture/08-api-contract.md) |
+| how to operate the live stack | [../OPERATIONS.md](OPERATIONS.md) |
+| per-artifact hyperparameters | [../TRAINING.md](TRAINING.md) |
+| what a third party can and cannot reproduce | [../REPRODUCIBILITY.md](REPRODUCIBILITY.md) |
+
+---
+
+> **Chapter summary.** Python 3.11+ and a CPU are sufficient; there is no CUDA requirement, and all
+> placement is `.to(device)`, never `.cuda()`. The single registry is `configs/base.yaml`, its hash
+> **`78f1e3700da15aa1`** is frozen, and **editing the config moves the hash and invalidates every artifact
+> keyed to it** — the one non-reversible action in the repository. `core/schemas.py` is the binding
+> contract and every specialist returns exactly `SpecialistResult`; the specialist interface is four
+> methods in `specialists/base.py`, and a new specialist is added by implementing it and adding a
+> `SpecialistSpec` row keyed on **capability**, not name. Tests run from the venv, targeted, because the
+> full suite trips the sandbox's bulk-delete guard. Four artifacts train locally; two — change-VQA and the
+> VLM LoRA — train on an external GPU and are not one-command reproducible. Eight development traps are
+> recorded, of which the most dangerous are the stale untracked `deploy/`, the annotation-scope trap that
+> turns a `Request` parameter into a required query parameter, and the Chrome focus trap that produces a
+> silent false pass. `NOT RUN`/`OPEN` items include B-07 and B-02 (both `OPEN`), the absent `LICENSE`, the
+> stale monorepo README and `hf/` docs, and several `UNKNOWN — not established from the available
+> evidence` gaps.