docs/TESTING.md

# SatQuery AI — Testing, Verification and Evidence Integrity

**Status of this document.** This is the testing chapter of the public SatQuery AI release. It enumerates
the actual suites that exist, states what each one pins, and reports the full-suite result **including its
failures**. Every test file, count, command and code excerpt below was read from the repository. Where a
value could not be established it is written
`UNKNOWN — not established from the available evidence`.

**Status vocabulary** (see `DOCS_STYLE_GUIDE.md` §2): `IMPLEMENTED` · `VERIFIED` · `MEASURED` ·
`ATTEMPTED` · `NOT RUN` · `BLOCKED` · `DEFERRED` · `REJECTED` · `OPEN` · `RESOLVED` · `CLOSED`.

**The one rule this chapter follows without exception:** a green suite is not evidence about a property
nobody wrote an assertion for, and a failure is never reclassified as a pass. The project learned this the
hard way twice — a guard that pinned a *defect* as the specification
(`docs/DEPLOYMENT_ARCHITECTURE.md` §5.5) and a guard whose assertion was satisfied by an upstream fix so it
could not see the downstream site (`F-15b`). Both are recorded below rather than smoothed over.

---

## Table of contents

- [0. How to read this document](#0-how-to-read-this-document)
- [1. Testing philosophy: tests pin contracts, not implementation](#1-testing-philosophy-tests-pin-contracts-not-implementation)
- [2. The suite inventory](#2-the-suite-inventory)
- [3. The doc-guard tests: keeping documentation and code in sync](#3-the-doc-guard-tests-keeping-documentation-and-code-in-sync)
- [4. The evidence engine: purity, determinism and the reproducibility property](#4-the-evidence-engine-purity-determinism-and-the-reproducibility-property)
- [5. The frontend live-wiring suite](#5-the-frontend-live-wiring-suite)
- [6. The doc/frontend suite](#6-the-docfrontend-suite)
- [7. The full `tests/unit` result, and the 5–6 environmental failures](#7-the-full-testsunit-result-and-the-56-environmental-failures)
- [8. The live-validation harness and its integrity discipline](#8-the-live-validation-harness-and-its-integrity-discipline)
- [9. Verdict-independent evidence: recomputing from raw records](#9-verdict-independent-evidence-recomputing-from-raw-records)
- [10. How to run the tests](#10-how-to-run-the-tests)
- [11. What is NOT tested](#11-what-is-not-tested)
- [12. Open items and evidence index](#12-open-items-and-evidence-index)

---

## 0. How to read this document

### 0.1 The three evidence classes

The repository encodes an evidence class on every test via a pytest marker (`pytest.ini`):

```ini
markers =
    unit: fast, no I/O, no server, no network. Evidence about a component in isolation.
    integration: exercises two or more real components wired together in-process.
    smoke: a minimal end-to-end path run against real local artifacts, not a mock.
    real_inference: ran the real model on real inputs in this environment.
```

The comment above the markers states the purpose exactly, and it is the reason the markers exist rather
than being decorative:

> *"The markers exist to make a test's EVIDENCE CLASS explicit: a unit test never claims a deployment was
> exercised, and no test may be reported as 'real inference' unless it ran the real model on real data in
> this environment."*

And one deliberate non-marker, quoted in full because it is a subtle but important choice:

> *"`environment_blocked` is not a test marker: a blocked path is reported in the STEP 7 report rather than
> encoded as a permanently-skipped test, because **a skip can be mistaken for coverage**."*

That sentence is the spine of this chapter. A skip that looks like coverage is the failure mode the marker
design is built to avoid.

### 0.2 What "verified" means here

| Term | Means | Example |
|---|---|---|
| **Unit** | one component, no I/O, no server, no network | `tests/unit/test_evidence_engine.py` |
| **Integration** | two or more real components wired in-process | `tests/integration/test_step7_backend_chain.py` |
| **Smoke** | a minimal end-to-end path over real local artifacts | `tests/unit/phase6_smoke.py` (standalone) |
| **Real inference** | the real model on real inputs, in this environment | — see §11 |
| **Live validation** | the deployed stack driven by a browser | §8 |

The distinction matters most at the boundary between **integration** and **live**. `docs/API_CONTRACT.md`
§8 states the boundary plainly: *"no test in this repository dials a network address"*, and it points at
`docs/ITEM5_INTEGRATION_SUITE_SCOPE.md`, which *"records what the 26-test integration suite proves (the
app's boundary, in-process) and what only a live deployment can prove (reachability, cold start, memory
ceilings)."* A green `tests/integration` run is therefore **not** evidence about a deployment.

### 0.3 The two defects that shaped the discipline

Two incidents are referenced repeatedly below, because they are why this chapter is written the way it is:

1. **A guard that pinned the defect.** The test
   `tests/unit/test_controller.py::test_trace_records_inputs_and_query` asserted
   `envelope.trace.inputs == list(request.assets)` — which, after the upload handler rewrote handles into
   paths, pinned a **filesystem-path disclosure as the specification**. The lesson, recorded in
   `docs/DEPLOYMENT_ARCHITECTURE.md` §5.3: *"A guard is only as strong as the behaviour it pins, and this
   one pinned the defect — a reminder that a green suite is not evidence about a property nobody wrote an
   assertion for."*
2. **A guard that could not fail.** The `to_trace()` seam's scrub had **no covering test**: falsifying it
   alone left the producer-side guard **GREEN**, because the producer had already scrubbed the value the
   guard inspected. Recorded as F-15b in `docs/DEPLOYMENT_ARCHITECTURE.md` §5.5, and remedied by
   `test_to_trace_scrubs_a_detail_that_arrives_from_any_constructor`, which constructs the entry
   **directly** through the constructor.

Both are the same shape: *a guard whose assertion is satisfied by something other than the code under
test.*

---

## 1. Testing philosophy: tests pin contracts, not implementation

### 1.1 The philosophy, as the suites state it

The repository's test docstrings state a consistent philosophy, and the consistency is the point. Four
representative statements, each quoted from a module docstring:

**Doc-guards pin examples, not prose** (`tests/unit/test_api_contract_doc.py`):

> *"These tests validate the EXAMPLES, not the prose. They cannot detect a wrong sentence (e.g. a
> mis-stated latency budget). What they guarantee is that a frontend developer who copies an example
> verbatim gets a request the server accepts -- which is the failure mode that actually costs time."*

**Guards pin the route, not the validity** (`tests/unit/test_frontend_live_wiring.py`):

> *"It also pins the `chang` word-boundary defect: `\bchang\b` cannot match 'changed', so the page's own
> default question ('What changed here?') routed to `vqa` instead of the change path. That is invisible to
> any test that only checks 'is this a valid enum value' — both branches produce a valid value. The test
> therefore asserts the ROUTE, not just the validity."*

**Tests read shipped source when there is no importable symbol** (`tests/unit/test_frontend_live_wiring.py`):

> *"Why these are read from the source text: the frontend is plain ES5 with no build step and no module
> exports, so there is no importable symbol. Parsing the source is the only way to assert the shipped
> bytes. The parsing is anchored on named tokens rather than line numbers, so it does not silently pass if
> the file is restructured."*

**A runbook must not assert facts the repository contradicts** (`tests/unit/test_runbook_doc.py`):

> *"A runbook is worse than useless if a copied command fails or a quoted number was never measured: the
> operator concludes the *system* is broken rather than the *document*."*

### 1.2 The three anti-patterns the suites are written to avoid

| Anti-pattern | What it looks like | How the suites avoid it |
|---|---|---|
| **Pinning the defect** | an assertion that encodes today's buggy behaviour as correct | guards are **falsified** before being trusted: the fix is reverted and the guard is shown to fail (§1.3) |
| **Vacuous guards** | an assertion satisfied by an upstream layer, so the downstream site is untested | the seam is tested by constructing the object **directly** through its constructor (F-15b) |
| **Measurement by prose** | a text search that is tripped by documentation *about* a defect | guards walk the **parsed** examples, not the document text (`test_api_contract_doc.py::test_no_contract_example_fabricates_an_artifact_uri`) |

The third is worth quoting in full, because it is a rule about what a guard must fail on
(`tests/unit/test_api_contract_doc.py`):

> *"This walks the PARSED examples rather than the document text on purpose. The prose in section 2.4
> deliberately quotes the old `artifact://run/9f2c.../change_map.png` example when explaining why it was
> removed, so a text search would be tripped by documentation *about* the removal -- which is measuring
> the wrong thing. A guard must fail on the defect, not on a description of the defect."*

### 1.3 Falsification is part of the discipline

A guard is not trusted until it has been shown to **fail** against the defect it guards. The pattern is
recorded with measurements throughout the project. The clearest instance is the F-15 four-site
falsification (`docs/DEPLOYMENT_ARCHITECTURE.md` §5.5), where each site was reverted individually and the
guards re-run:

| Site reverted | `test_a_construction_failure_publishes_no_filesystem_path` | `test_to_trace_scrubs_a_detail_that_arrives_from_any_constructor` |
|---|---|---|
| `registry.py:597` (`_failure_entry`) | **FAILS** | passes *(correctly — it tests the seam, not the producer)* |
| `registry.py:309` (`to_trace`) | **passes — blind** | **FAILS** |

The "passes — blind" cell is the F-15b finding: the producer guard cannot see the seam, because the
producer has already scrubbed the value. The two guards are therefore **not duplicates**, and the table is
the evidence.

A second falsification table covers the two controller sites (`docs/DEPLOYMENT_ARCHITECTURE.md` §5.5):

| Site reverted | `test_a_failed_step_publishes_no_filesystem_path` — assertion that fires |
|---|---|
| `controller.py:801` (`_failure_evidence`) | line 590, `payload["message"]` |
| `controller.py:969` (`_warnings`) | line 582, `warnings[]` |

And a third, for the trace-path fix (`docs/DEPLOYMENT_ARCHITECTURE.md` §5.4):

| Step | Result |
|---|---|
| Revert only the `PARSE` site, run the module | **1 failed, 55 passed** at `tests/unit/test_controller.py:1153` |
| Restore byte-exact, verify hash | `0c00ae6f54300668122728dc706344044bc19d48328ceead30323c7147e04397` |
| Module re-run | **56 passed** |
| Is the guard vacuous? | **No** — the shape check only discriminates because the fixture writes real GeoTIFFs, and the name-equality check rejects the revert independently |

The "restore byte-exact, verify hash" row is the anti-tamper step: after reverting a site to falsify a
guard, the file is restored and its hash checked, so the falsification itself cannot leave the tree in a
modified state.

### 1.4 Determinism is asserted, not hoped for

Several suites pin **determinism** as a first-class property rather than a nice-to-have:

| Suite | Determinism property |
|---|---|
| `tests/unit/test_evidence_engine.py` | the same claims produce a byte-identical, identically-ordered collection even when uuids differ between processes (§4) |
| `tests/unit/test_router_threshold_sweep.py` | validation-only sweep contracts |
| `tests/unit/test_change_threshold_sweep.py` | validation-only sweep contracts |
| `tests/unit/test_report_generator.py` | *"pure, deterministic, fixtures only"* |
| `tests/unit/test_vlm_evaluate.py` | the predeclared acceptance rule |
| `tests/unit/test_fusion_seed_variance.py` | the predeclared seed-variance pass |

The determinism claim that matters most is the evidence engine's, because it is what makes the live
validation's verdict-recomputation property hold (§9).

---

## 2. The suite inventory

### 2.1 The layout

`pytest.ini` sets the roots:

```ini
[pytest]
testpaths = tests
pythonpath = .
addopts = -q --tb=short
filterwarnings =
    ignore::DeprecationWarning
    ignore::PendingDeprecationWarning
    ignore::rasterio.errors.NotGeoreferencedWarning
```

`tests/` contains eight directories and two top-level modules:

| Location | Contents | Evidence class |
|---|---|---|
| `tests/unit/` | 98 Python files — 94 collected test modules + `__init__.py`, `conftest.py`, `_qa_probe_tokens.py`, `phase6_smoke.py` | `unit` (mostly) |
| `tests/e2e/` | `test_demo.py`, `test_gate4_e2e.py`, `_demo_probe.py` | end-to-end |
| `tests/geospatial/` | `test_transform.py` | `unit` |
| `tests/integration/` | `test_step7_backend_chain.py`, `test_step7_r02_serving.py` | `integration` |
| `tests/leakage/` | `test_leakage.py` | `unit` |
| `tests/model/` | `test_vlm_contract.py`, `test_vqa_quality_gate.py` | `unit` / `integration` |
| `tests/routing/` | `test_router.py` | `unit` |
| `tests/` | `test_config.py`, `test_schemas.py` | `unit` |

`tests/unit` totals **49,673 lines** across its 98 Python files. Two files there are **deliberately not
collected**:

| File | Why it is not collected |
|---|---|
| `tests/unit/_qa_probe_tokens.py` | *"QA scratch probe (NOT collected by pytest -- underscore prefix)."* |
| `tests/unit/phase6_smoke.py` | *"Phase 6 real CPU smoke test -- standalone, NOT collected by pytest."* |
| `tests/unit/conftest.py` | *"Shared fixtures for the Phase 6 VLM unit tests."* — a fixture module, not a test module |
| `tests/unit/__init__.py` | package marker |

### 2.2 The full file inventory, with what each covers

The descriptions below are the **first line of each module's own docstring** — the module's statement of
what it pins, not a paraphrase.

#### 2.2.1 Gateway, deployment and configuration

| File | What it covers (module docstring) |
|---|---|
| `test_gateway_app.py` | *"The gateway ASGI layer: its contract, its allowlist, and real HTTP execution."* |
| `test_gateway_assets.py` | *"The ephemeral asset store: opaque handles, three obligations, real lifetimes."* |
| `test_gateway_policy.py` | *"The gateway's decisions must be correct, cheap, and never reach the Space on a rejection."* |
| `test_gateway_responsibilities.py` | *"STEP 7 §C — the eighteen gateway responsibilities, as UNIT tests."* |
| `test_space_app.py` | *"The Space entrypoint must be cheap, honest, and hash-preserving."* |
| `test_deployment_adapter.py` | *"The deployment capability adapter: registry-authoritative, contract-shaped."* |
| `test_deploy_config.py` | *"Tests for the Phase 18 deploy packaging manifest and its validator."* |
| `test_deploy_requests.py` | *"Phase 18 §79 DEPLOYMENT: cold-start and sequential-request behaviour."* |
| `test_config.py` | *"Phase 1 tests — configuration registry and frozen contract guards."* |
| `test_code_revision.py` | *"Tests for `core.code_revision` — provenance defect section 6b."* |
| `test_packaging_script.py` | *"Regression tests for `scripts/package_kaggle_code.py`."* |

#### 2.2.2 Controller, planner, registry and schemas

| File | What it covers |
|---|---|
| `test_controller.py` | *"Tests for `core.controller` — dispatch, partial failure and assembly (Phase 15)."* |
| `test_controller_seam.py` | *"The controller's asset-resolution seam — regression tests for a real defect."* |
| `test_planner.py` | *"Tests for `core.planner` — the deterministic policy planner (Phase 15)."* |
| `test_registry.py` | *"Tests for `core.registry` — the specialist capability registry (Phase 15)."* |
| `test_schemas.py` | *"Phase 1 tests — canonical schema contract."* |
| `test_errors.py` | *"Tests for `core.errors` — the taxonomy, and the F-15 path scrubber."* |
| `test_evidence_engine.py` | *"Tests for the evidence engine — the aggregator (Phase 13)."* (§4) |
| `test_report_generator.py` | *"Unit tests for `reports.generator` — pure, deterministic, fixtures only."* |

#### 2.2.3 Router

| File | What it covers |
|---|---|
| `test_router_threshold_sweep.py` | *"Phase 4/13 — contracts for the validation-only ROUTER threshold sweep."* |
| `tests/routing/test_router.py` | *"Phase 4 tests — the intent router."* |

#### 2.2.4 Change detection and change-VQA (the largest subsystem by file count)

| File | What it covers |
|---|---|
| `test_change.py` | *"Phase 9 tests — change detection model, dataset, and post-processing."* |
| `test_change_levir_real_layout.py` | *"Phase 9 — the LEVIR-CD loader against the REAL dataset layout."* |
| `test_change_phase9_decontamination.py` | *"Phase 9 -- pins that stop the change-detection module from being re-contaminated."* |
| `test_change_specialist.py` | *"Phase 9 — the change specialist serving contract."* |
| `test_change_threshold_sweep.py` | *"Phase 9 — contracts for the validation-only threshold sweep."* |
| `test_change_train_script_contract.py` | *"Phase 9 — contracts for `scripts/train_change.py`."* |
| `test_change_vqa_amp.py` | *"R-02 — the mixed-precision training contract."* |
| `test_change_vqa_dataset.py` | *"R-02 test areas E-J — the CDVQA dataset layer."* |
| `test_change_vqa_eval_cli.py` | *"R-02 — the evaluation CLI's empty-split diagnosis."* |
| `test_change_vqa_features.py` | *"R-02 test areas K-L — frozen change features and the feature caches."* |
| `test_change_vqa_head.py` | *"R-02 test areas M-Q — batching, the reasoning head, and the metrics."* |
| `test_change_vqa_integration.py` | *"R-02 test area R — the change-VQA specialist and its planner/registry wiring."* |
| `test_change_vqa_kaggle_notebook.py` | *"R-02 — the Kaggle notebook's discovery contract."* |
| `test_change_vqa_prepare_script.py` | *"R-02 — `prepare_record.json` must be the provenance of the whole directory."* |
| `test_change_vqa_promotion.py` | *"STEP 1 — the promoted R-02 change-VQA head at its canonical serving path."* |
| `test_change_vqa_smoke.py` | *"R-02 — the local smoke training run, and the artifact it leaves behind."* |
| `test_change_vqa_vocab.py` | *"R-02 test areas A-D — the CDVQA answer space and question ontology."* |
| `test_cdvqa_adapter.py` | *"Tests for the CDVQA reader, join and example construction."* |
| `test_cdvqa_second_overlap.py` | *"Tests for scripts/check_cdvqa_second_overlap.py."* |

#### 2.2.5 Optical-SAR

| File | What it covers |
|---|---|
| `test_optical_sar_croma.py` | *"CROMA wrapper — the interface verified against the real model."* |
| `test_optical_sar_croma_geometry.py` | *"Phase 11 geometry guards — the four requirements that were *measured but…"* |
| `test_optical_sar_fusion_head.py` | *"Optical-SAR fusion head — the frozen concatenation and channel dropout."* |
| `test_optical_sar_prompts.py` | *"Optical-SAR prompts — the narration boundary."* |
| `test_optical_sar_radiometry.py` | *"CROMA encoder-input radiometry — the DEV-2 ruling, as executable guards."* |
| `test_optical_sar_sensor_adapter.py` | *"Optical-SAR sensor adapter — the two hard rules, enforced."* |
| `test_optical_sar_specialist.py` | *"Phase 11/12 — the optical-SAR specialist serving contract."* |
| `test_app_serving_optical_sar.py` | *"Regression guards for the optical-SAR serving wiring (Phase 14 / Pass 16)."* |
| `test_eval_fusion_115.py` | *"Tests for the pre-registered 11.5 metric tool."* |
| `test_fusion_extraction.py` | *"Tests for the Phase 12 frozen-feature extraction pipeline (fixtures only)."* |
| `test_fusion_seed_variance.py` | *"Tests for the predeclared seed-variance pass (Gate F D-03 / D-05)."* |
| `test_fusion_training.py` | *"Tests for the Phase 12 fusion-head training loop (fixtures only)."* |
| `test_reben_adapter.py` | *"Tests for the reBEN v2 -> PairedSample adapter (Phase 12)."* |
| `test_analyze_reben_labels.py` | *"Tests for the reBEN label analyser."* |

#### 2.2.6 Grounding

| File | What it covers |
|---|---|
| `test_grounding_dataset.py` | *"Tests for the Phase 8 grounding dataset."* |
| `test_grounding_degenerate_fallback.py` | *"Regression tests for the degenerate fallback-box guard (Phase 8)."* |
| `test_grounding_feature_extraction.py` | *"Tests for the frozen-feature cache."* |
| `test_grounding_head_wiring.py` | *"Wiring the trained RemoteCLIP grounding head into production (work order §5)."* |
| `test_grounding_metrics.py` | *"Tests for the grounding metrics."* |
| `test_grounding_nms.py` | *"Unit tests for grounding NMS (specialists/grounding/head.py::nms)."* |
| `test_vrsbench_loader.py` | *"Tests for the VRSBench referring-expression loader."* |
| `test_vrsbench_lazy_images.py` | *"Tests for the lazy-image mode of the VRSBench loader."* |

#### 2.2.7 VLM training and evaluation

| File | What it covers |
|---|---|
| `test_vlm_artifact.py` | *"Tests for the Phase 6 adapter artifact (`training.vlm.artifact`)."* |
| `test_vlm_collate.py` | *"Tests for Phase 6 batch assembly (`training.vlm.collate`)."* |
| `test_vlm_config.py` | *"Tests for the Phase 6 training recipe (`training.vlm.config`)."* |
| `test_vlm_dataset.py` | *"Tests for the Phase 6 BigEarthNet -> SmolVLM instruction corpus."* |
| `test_vlm_evaluate.py` | *"Tests for the predeclared acceptance rule (`training.vlm.evaluate`)."* |
| `test_vlm_evaluate_cache.py` | *"Tests for the resumable per-question answer cache in `training.vlm.evaluate`."* |
| `test_vlm_evaluate_decision_split.py` | *"Tests for `decide_acceptance(..., decision_split=...)`."* |
| `test_vlm_formatting.py` | *"Tests for Phase 6 prompt construction and loss masking (`training.vlm.formatting`)."* |
| `test_vlm_lora.py` | *"Tests for Phase 6 LoRA injection (`training.vlm.lora`)."* |
| `tests/model/test_vlm_contract.py` | *"Phase 5 tests — SmolVLM contract, prompts, and the VQA specialist."* |
| `tests/model/test_vqa_quality_gate.py` | *"Integration tests: the F5-5 quality gate blocks the VLM before it runs."* |

#### 2.2.8 Datasets, metrics and evaluation

| File | What it covers |
|---|---|
| `test_bigearthnet_blocks.py` | *"Tests for the T2 spatial-block scene key (Phase 12)."* |
| `test_bigearthnet_pairing.py` | *"Tests for the BigEarthNet optical<->SAR pairing step (Phase 12)."* |
| `test_bigearthnet_prep.py` | *"Tests for the BigEarthNet reader and instruction-pair construction."* |
| `test_prepare_script.py` | *"Tests for the BigEarthNet preparation CLI."* |
| `test_caption_metrics.py` | *"Caption metrics (ruling R-16): BLEU, ROUGE-L, CIDEr, BERTScore."* |
| `test_vqa_metrics.py` | *"Tests for the Tier-1 VQA / Change-VQA text metrics (`evaluation.metrics.vqa`)."* |
| `test_eval_normalize.py` | *"Tests for `evaluation.normalize` — the plan section 63 metric normaliser."* |
| `test_evaluation_runner.py` | *"STEP 5 — the evaluation runner."* |
| `test_calibration.py` | *"STEP 2 — the calibration fitter, the artifact, and the consumer wiring."* |
| `test_quality_gate.py` | *"Tests for the deterministic input-quality gate (finding F5-5)."* |
| `test_benchmark_adapters.py` | *"STEP 5 — benchmark adapters: contract, registry, and the non-invention guard."* |
| `test_benchmark_adapters_real.py` | *"The four real-corpus benchmark adapters: correctness against synthetic fixtures."* |
| `test_public_test_corpus.py` | *"STEP 5 — the immutable public-test corpus (plan section 37, Test Set Firewall)."* |
| `test_manifest_freeze.py` | *"Tests for the frozen dataset-manifest record (`evaluation/manifest_freeze.json`)."* |
| `test_prompt_freeze.py` | *"Tests for the frozen prompt-set record (`evaluation/prompt_freeze.json`)."* |
| `test_run_manifest_population.py` | *"Tests for STEP 4 — runtime population of the run manifest."* |
| `test_provenance_fixes.py` | *"Tests for the R-02 provenance fixes: sections 6a, 6b and 6c."* |
| `test_diagnose_feature_cache.py` | *"Tests for `scripts/diagnose_feature_cache.py`'s exit-code contract."* |

#### 2.2.9 Serving composition and the app boundary

| File | What it covers |
|---|---|
| `test_app_serving.py` | *"Regression tests for `app.serving` — the public serving composition root."* |
| `test_phase6_closure.py` | *"Tests for the Phase 6 closure record."* |

#### 2.2.10 Contract-conformance, frontend and documentation guards

| File | What it covers |
|---|---|
| `test_api_contract_doc.py` | *"The API contract must not drift from the schemas it documents."* |
| `test_frontend_guide_doc.py` | *"The contract documents must stay consistent with each other and the schemas."* |
| `test_frontend_live_wiring.py` | *"Regression tests for the frontend <-> orchestrator live wiring."* (§5) |
| `test_runbook_doc.py` | *"The deployment runbook must not assert facts the repository contradicts."* |
| `test_deploy_config.py` | *"Tests for the Phase 18 deploy packaging manifest and its validator."* |
| `test_step7_api_contract_conformance.py` | *"STEP 7 section D -- the API contract, tested against the schema that implements it."* |
| `test_safe_delete_shim.py` | *"The Windows safe-delete shim: verbatim prefixes, and the shell's return code."* (§7) |

#### 2.2.11 The other directories

| File | What it covers |
|---|---|
| `tests/geospatial/test_transform.py` | *"Phase 2 tests — geospatial transforms and the raster input contract."* |
| `tests/leakage/test_leakage.py` | *"Phase 3 tests — Gate 1: dataset manifests and scene-level leakage isolation."* |
| `tests/integration/test_step7_backend_chain.py` | *"STEP 7 sections H, I, M -- the real inference service, driven over HTTP."* |
| `tests/integration/test_step7_r02_serving.py` | *"STEP 7 section I -- the R-02 head, verified through the real serving path."* |
| `tests/e2e/test_demo.py` | *"End-to-end tests for the Phase-19 demonstration driver (`demo.run_demo`)."* |
| `tests/e2e/test_gate4_e2e.py` | *"Gate-4 end-to-end test — the full control tier in its real local state."* |
| `tests/e2e/_demo_probe.py` | *"Subprocess probe: run the demo in a FRESH interpreter and report which…"* (helper) |

### 2.3 What the inventory says about coverage shape

Three observations a reader should draw from the inventory:

1. **The largest single cluster is change detection and change-VQA** (19 files), which matches the
   project's stated engineering priority ordering (`docs/MASTER_ARCHITECTURE_PLAN.md` §2.1: optical-SAR
   first, change second).
2. **Every capability has a *serving contract* test, separate from its *model* tests.** For example
   `test_change.py` covers the model/dataset/post-processing while `test_change_specialist.py` covers
   *"the change specialist serving contract"*; the same split exists for optical-SAR
   (`test_optical_sar_fusion_head.py` vs `test_optical_sar_specialist.py`) and grounding. The split is
   deliberate: a model that works in a notebook and a specialist that serves the contract are different
   claims.
3. **Documentation is tested as code.** Five of the 94 modules are doc-guards (§3), and they are counted in
   the 183-passed doc/frontend suite (§6) rather than being an afterthought.

### 2.4 The integration suite's documented scope

`docs/ITEM5_INTEGRATION_SUITE_SCOPE.md` is the record of what the integration suite proves. The contract
summarises it (`docs/API_CONTRACT.md` §8):

> *"`docs/ITEM5_INTEGRATION_SUITE_SCOPE.md` records what the 26-test integration suite proves (the app's
> *boundary*, in-process) and what only a live deployment can prove (reachability, cold start, memory
> ceilings). Read it before treating a green `tests/integration` run as evidence about a deployment —
> **no test in this repository dials a network address**, including this contract's own `/v1/*`
> examples."*

The exact number of tests **collected** in a single `tests/integration` invocation is
`UNKNOWN — not established from the available evidence`. The documented figure is 26; the evidence this
chapter can establish is the two module files and their stated scope.

---

## 3. The doc-guard tests: keeping documentation and code in sync

### 3.1 Why documentation is tested

Five modules treat documentation as a **tested artefact**. The rationale is stated in each module's
docstring, and the common thread is that a documentation defect costs a real debugging session:

| Guard | The failure it prevents |
|---|---|
| `test_api_contract_doc.py` | a frontend builds against an example that does not validate, and the cause looks like a frontend bug |
| `test_frontend_guide_doc.py` | a value a frontend must send or read is documented wrong, producing a `422` and a wasted session |
| `test_runbook_doc.py` | an operator copies a command that fails, or quotes a number that was never measured, and concludes the *system* is broken |
| `test_deploy_config.py` | the deploy manifest silently moves the frozen `Config.hash` |
| `test_step7_api_contract_conformance.py` | the contract and the schema that implements it drift apart |

`test_api_contract_doc.py` records what the guards actually catch, with a list of four real defects the
act of writing the contract produced:

> *"Writing this contract produced exactly that failure four times over -- `Box` fields were nested instead
> of flat, `Evidence` used `summary`/`value`/`source` instead of
> `coordinates`/`score`/`source_specialist`, `CoordinateSystem` values were shorthand rather than the real
> `normalized_0_1`/`pixel`/`geo`, and `ExecutionTrace.inputs`/`outputs` were dicts rather than lists. Each
> was caught by validating the examples against the real models, which is what these tests do
> permanently."*

### 3.2 `test_api_contract_doc.py`

**Scope:** validate every fenced ```json block in `docs/API_CONTRACT.md` against the real Pydantic models.

| Mechanism | Detail |
|---|---|
| Fixture `contract_text` | asserts the document exists, then reads it |
| Fixture `json_blocks` | regex-extracts every ```json block and parses it; a block that is not valid JSON raises `AssertionError` naming the block index |
| `_find(blocks, predicate)` | locates an example by shape rather than by position |

| Test | What it pins |
|---|---|
| `test_every_json_block_in_the_contract_parses` | *"A malformed example is unusable; this fails loudly rather than silently."* |
| `test_the_result_envelope_example_validates` | the envelope example validates against `ResultEnvelope`; `schema_version == SCHEMA_VERSION`; `confidence.method in ("uncalibrated", "temperature_scaling")` |
| `test_no_contract_example_fabricates_an_artifact_uri` | no parsed example contains an `artifact://` URI or a filesystem path in `change_map` / `artifact_ref` (walks the **parsed** examples, §1.2) |
| (further tests) | documented task/coordinate-system values are the real ones; every error code documented; the frozen config hash is recorded; the calibration caveat is recorded |

The `confidence.method` assertion is the honest-signal guard: it pins that the contract's example cannot
claim a calibration method the engine does not have. §4 shows the engine side of the same rule.

### 3.3 `test_frontend_guide_doc.py`

**Scope:** `docs/API_CONTRACT.md` and `docs/FRONTEND_INTEGRATION.md` must agree with each other **and** with
`core/schemas.py`.

It imports the real models — `Box, CoordinateSystem, Evidence, EvidenceType, Region, Task` — and compares
the prose against them. The docstring lists the five defects that motivated it:

> *"Writing them produced, in order: `CoordinateSystem` documented as `normalized`/`geographic` (real:
> `normalized_0_1`/`geo`), `Box` geometry documented as nested (real: flat), `Evidence` fields named
> `summary`/`value`/`source` (real: `coordinates`/`score`/`source_specialist`),
> `ExecutionTrace.inputs`/`outputs` as objects (real: lists), and `Task.unsupported` missing from both
> tables."*

The representative test asserts a **bidirectional** requirement:

```python
def test_every_task_is_documented_in_both_documents(api_text, frontend_text):
    for task in Task:
        assert f"`{task.value}`" in api_text, f"Task.{task.value} missing from API contract"
        assert f"`{task.value}`" in frontend_text, (
            f"Task.{task.value} missing from the frontend guide; a value the client "
            f"may receive but cannot find documented is a crash waiting to happen"
        )
```

The message on the second assertion is the rationale: *"a value the client may receive but cannot find
documented is a crash waiting to happen."*

The suite also pins: no login screen, no secrets in the browser, the confidence-property trap, and
serialized analyses (`docs/EVALUATION.md` §7.2).

### 3.4 `test_runbook_doc.py`

**Scope:** `docs/BACKEND_DEPLOYMENT_RUNBOOK.md` must not assert facts the repository contradicts.

The module names **two failure modes**:

> 1. *"**Invented measurements.** The runbook quotes command output. If it quotes a byte size, a
>    capability list, a function name or an exit code that the repository does not produce, the quote is
>    fabricated. Where a fact is cheap to verify locally, this module verifies it."*
> 2. *"**Invented API surface.** The runbook calls into the codebase (`build_serving_registry`,
>    `available()`, `CHANGE_CHECKPOINT`, `CHANGE_VQA_HEAD`). A runbook that names a function the code does
>    not export is a runbook whose commands raise `AttributeError` on the operator's first copy-paste."*

It reads three documents — the runbook, `docs/DEPLOYMENT_ARCHITECTURE.md` and `docs/API_CONTRACT.md` — so
that a claim contradicted *across* documents is caught, not only a claim contradicted by code.

**The most important class of assertion in this module** is stated in its docstring:

> *"The absence of a deployment cannot be tested here, so these tests assert the runbook's *local* claims
> and, separately, assert that the document itself labels its unimplemented sections as unimplemented. The
> second class matters more than it looks: a runbook that describes a live deployment without having
> performed one is the exact overclaim the work order forbids."*

So the guard pins **honesty about status**, not merely factual accuracy: an "undecided" item must be
declared undecided, and a "designed" item must not be described as "verified" (`docs/EVALUATION.md` §7.2).

### 3.5 `test_deploy_config.py`

**Scope:** the Phase 18 deploy manifest and its validator, and the **config-hash regression guard**.

The docstring states the two properties it proves:

> *"These tests prove the manifest is INERT — it is never merged into the config registry and cannot move
> `Config.hash` — and that the validator rejects each failure mode. Negative cases use in-memory dicts via
> `validate_documents`, so the real config files are never mutated."*

Key elements:

| Element | Detail |
|---|---|
| `REGISTRY_MARKER` | `test_deploy_manifest_exists_and_declares_the_non_registry_marker` asserts `doc[REGISTRY_MARKER] is False` |
| `test_validator_reports_no_errors_on_the_real_files` | `assert validate() == []` |
| `test_validator_cli_exits_zero_as_a_standalone_script` | drives the real entry point **in a child interpreter** (the same one running the suite) *"to prove the module's own `sys.path` bootstrap makes `core` importable WITHOUT pytest's `pythonpath = .` shortcut"* |
| `GPU_DURATION_KEYS` | `gpu_duration_vqa`, `gpu_duration_grounding`, `gpu_duration_change`, `gpu_duration_optical_sar` |
| `FROZEN_CONFIG_HASH` | imported from `tests.test_config` rather than re-declared — *"We import it rather than re-declare it, so the two cannot drift."* |

The `FROZEN_CONFIG_HASH` import is the anti-drift move applied to the guard itself: the deploy-config test
does not carry its own copy of `78f1e3700da15aa1`, it imports the authority.

The suite also pins the C-8 constraints: `torch_compile` false, CPU mode required, lazy load, single model
cache (`docs/EVALUATION.md` §7.2).

### 3.6 `test_step7_api_contract_conformance.py`

*"STEP 7 section D -- the API contract, tested against the schema that implements it."* This is the
schema-level counterpart to `test_api_contract_doc.py`: where the doc-guard validates the *examples* in the
markdown, this module tests the *contract* against the implementing schema directly.

### 3.7 What the doc-guards cannot do

The modules say so themselves, and the honesty is worth reproducing because it bounds what a green run
means:

> *"Prose can still be wrong in ways these tests cannot see -- a mis-stated latency, a wrong reason for a
> decision. What is asserted is that every value a frontend must send or read is the real value, which is
> the class of error that produces a 422 and a wasted debugging session."*
> — `tests/unit/test_frontend_guide_doc.py`

> *"These tests validate the EXAMPLES, not the prose. They cannot detect a wrong sentence (e.g. a
> mis-stated latency budget)."*
> — `tests/unit/test_api_contract_doc.py`

So a doc-guard guarantees **example-level and value-level** conformance. It does not guarantee that a
sentence is true. That distinction is the reason `test_runbook_doc.py` exists separately — it targets a
different class of claim (numbers, names, and status honesty) that the example-validators cannot reach.

---

## 4. The evidence engine: purity, determinism and the reproducibility property

### 4.1 Why this suite exists at all

The evidence engine is the component that turns a `SpecialistResult` into a *proof*: an ordered,
identified, digestible `EvidenceCollection`. Everything the release claims about "evidence" downstream —
the trace, the audit trail, the ability to recompute a verdict from a recorded run — rests on one
property of this component: **the same inputs must produce the same output, byte for byte.**

If that property fails, then two runs of the same request can produce two different evidence sets, and
no verdict can be recomputed from a record. The evidence engine is therefore the load-bearing component
behind §9's "verdict-independent evidence" claim, and its suite is the guard on that load-bearing
property.

The suite is `tests/unit/test_evidence_engine.py` — **73 test functions** in 877 lines
(`tests/unit/test_evidence_engine.py`).

### 4.2 The contract, in the module's own words

The module docstring states exactly two failure modes it is written against, and a third honesty rule:

> *"The engine's whole value is that it is *reproducible* and *lossless*. So the tests are mostly about
> the two ways that can silently break:*
>
> *determinism — the same claims must produce the same ordered, identified collection even when
> specialists finish in a different order, or when the uuids differ between processes.*
> *no loss — no specialist's evidence may vanish without being counted, and agreement between
> specialists must be recorded rather than collapsed with one name thrown away.*
>
> *Plus the honesty rule on confidence: an uncalibrated engine must pass the raw score through and SAY
> it is uncalibrated, never manufacture a fitted mapping."*
> — `tests/unit/test_evidence_engine.py`

Three properties, then: **determinism**, **no loss**, and **confidence honesty**. The suite is
organised around them.

### 4.3 The determinism family

The core purity claim is a single test whose docstring is the whole argument:

```python
def test_aggregate_is_deterministic_across_repeat_calls() -> None:
    """Same input -> byte-identical output. The core purity claim."""
```

It asserts three things across two calls with the same input — identical `ids()`, identical
`evidence_digest()`, and identical `model_dump()` lists. "Byte-identical output" is not a figure of
speech: the digest is the serialised form, so a difference anywhere in the collection changes the digest.

The determinism family, read from the file:

| Test | What it pins |
|---|---|
| `test_aggregate_is_deterministic_across_repeat_calls` | the core purity claim (line 132) |
| `test_order_is_independent_of_input_order` | forward vs. backward input order give the same digest |
| `test_equal_scores_order_deterministically_by_coordinates` | *"Ties must not fall back to insertion order — that is input-order dependence wearing a disguise"* |
| `test_higher_score_sorts_first_within_a_type` | the ordering rule inside an evidence type |
| `test_unscored_items_sort_after_scored_items` | a total order even when some items carry no score |
| `test_types_are_grouped_together` | type grouping is stable |
| `test_ids_are_sequential_and_zero_padded` | the identifier format |
| `test_ids_restart_from_one_for_each_aggregation` | ids are a function of the collection, not global state |
| `test_source_results_are_not_mutated` | purity — computing evidence does not write back |
| `test_result_objects_are_not_mutated` | purity, at the result level |

The two mutation tests are the *purity* half of "purity and determinism": the engine is a function, not
a transformer of its inputs. A caller's `SpecialistResult` is the same object after `aggregate` as
before.

### 4.4 The no-loss family

Losslessness is the harder property to test, because the ways evidence can vanish are subtle. The
deduplication tests are where it is pinned:

| Test | What it pins |
|---|---|
| `test_payload_only_differences_are_deduplicated_as_one_claim` | two items differing only in payload are one claim |
| `test_identical_items_from_one_specialist_collapse` | intra-specialist dedup |
| `test_deduplication_ignores_random_uuid` | dedup is on content, not on the process-local uuid |
| `test_deduplication_ignores_payload_differences_and_merges_them` | payloads merge rather than one being dropped |
| `test_different_coordinates_are_not_deduplicated` | dedup does not over-collapse |
| `test_different_coordinate_systems_are_not_deduplicated` | a normalised box and a pixel box are different claims |
| `test_same_claim_from_two_specialists_is_kept_and_marked_corroborated` | **agreement is recorded, not collapsed** — the second specialist's name is kept as corroboration |
| `test_unrelated_items_carry_no_corroboration_marker` | the marker is meaningful, not always-on |
| `test_multiple_results_aggregate_without_loss` | the headline no-loss assertion |
| `test_aggregation_does_not_double_count_a_specialists_own_evidence` | lossless ≠ duplicating |

The corroboration tests are the direct expression of the docstring's *"agreement between specialists
must be recorded rather than collapsed with one name thrown away."* This is a design decision encoded
as a test: when two specialists agree, the collection records **both** names and marks the claim
corroborated, rather than keeping one and discarding the other.

The cap tests pin that truncation is *counted*, not silent:

| Test | What it pins |
|---|---|
| `test_zero_max_items_is_rejected` | an invalid cap is refused |
| `test_cap_is_respected_and_the_drop_is_counted` | items past the cap are dropped **and counted** |
| `test_no_truncation_flag_when_under_the_cap` | the truncation flag is truthful |
| `test_default_cap_matches_the_config_value` | `DEFAULT_MAX_ITEMS` tracks the config |
| `test_sources_are_reported_before_the_cap_is_applied` | *which* specialists contributed is recorded before truncation |

That last test is a specific honesty property: if the cap drops a specialist's only item, the collection
still records that the specialist contributed. Losslessness at the *source* level survives truncation at
the *item* level.

The empty/degenerate cases are pinned too: `test_empty_input_yields_an_empty_collection`,
`test_a_result_with_no_evidence_contributes_nothing`, `test_accepts_a_single_result_without_wrapping`,
and the two argument-error tests `test_both_results_and_evidence_is_rejected` /
`test_neither_results_nor_evidence_is_rejected`.

### 4.5 The confidence honesty rule

This is where the project's style guide (`DOCS_STYLE_GUIDE.md` §3) meets a test. The rule the engine
must obey: an uncalibrated engine **passes the raw score through and says it is uncalibrated**; it never
manufactures a fitted mapping. The tests that pin this:

| Test | What it pins |
|---|---|
| `test_no_calibration_passes_raw_through_and_says_so` | the default is identity + an explicit "uncalibrated" label |
| `test_uncalibrated_is_never_labelled_as_fitted` | the label cannot be silently upgraded |
| `test_identity_temperature_is_reported_as_uncalibrated` | `T = 1.0` is *not* calibration |
| `test_engine_without_calibration_does_not_claim_calibration` | the engine-level claim |
| `test_raw_outside_unit_range_is_clamped_not_rejected` | a robustness decision, pinned |
| `test_non_finite_raw_is_refused` | NaN/Inf are refused, not clamped |

This matters because the project's calibration genuinely made the metric **worse** (ECE
`0.013755 → 0.014929`, `DOCS_STYLE_GUIDE.md` §3) and is retained only because it is in the frozen
config. The engine must therefore never present calibration as an improvement, and these tests are what
enforce that at the code level. The `METHOD_TEMPERATURE` / `METHOD_UNCALIBRATED` constants are the two
labels, and `test_uncalibrated_is_never_labelled_as_fitted` is the guard that the first is never used
for the second.

The fitted path is tested separately and is deterministic:
`test_temperature_scaling_is_applied_when_fitted`,
`test_temperature_above_one_softens_and_below_one_sharpens`,
`test_temperature_scaling_is_monotonic`, `test_temperature_scaling_is_deterministic`,
`test_endpoint_scores_do_not_produce_nan`, `test_invalid_temperatures_are_rejected`, and
`test_specialist_degradation_survives_calibration` / `test_specialist_components_are_preserved_through_calibration`.

### 4.6 Calibration artifact I/O

The engine reads a calibration artifact from disk, and the tests pin the failure modes explicitly rather
than letting them degrade silently:

| Test | What it pins |
|---|---|
| `test_artifact_round_trips_through_json` | write→read is lossless |
| `test_nested_artifact_form_is_accepted` | both artifact shapes are accepted |
| `test_missing_artifact_degrades_instead_of_failing` | a missing artifact is a *degradation* by default |
| `test_missing_artifact_raises_when_required` | …but raises when the caller says it is required |
| `test_malformed_artifact_is_not_silently_swallowed` | a corrupt artifact is an error, not a shrug |
| `test_artifact_without_a_temperature_is_rejected` | the required field is required |
| `test_load_calibration_respects_the_master_switch` | the config switch is honoured |
| `test_load_calibration_reads_the_configured_file` | the configured path is the one read |
| `test_load_calibration_without_a_file_configured_is_none` | no config ⇒ `None`, not a guess |
| `test_engine_from_config_uses_the_configured_cap` / `..._falls_back_to_the_default_cap` | the cap wiring |

The pairing of `degrades_instead_of_failing` with `raises_when_required` is the pattern this repository
uses throughout: a soft default for a convenience path, and a hard error for the path where silence
would be a lie.

### 4.7 The scratch-root workaround, and why it is documented in the test file

The suite does **not** use pytest's built-in `tmp_path`. The module says why, and the reason is a
sandbox property, not a preference:

> *"Scratch root, repo-local. The built-in `tmp_path` fixture cannot finalize under this machine's
> sandbox, so artifact tests use this instead. It is the same directory `--basetemp=.pytest_tmp` points
> pytest at, so nothing is written outside the workspace."*
> — `tests/unit/test_evidence_engine.py`

```python
SCRATCH_ROOT = Path(__file__).resolve().parents[2] / ".pytest_tmp"
```

And the module docstring carries the operational requirement:

> *"The `--basetemp=.pytest_tmp` flag is mandatory (see the project handoff): the default pytest temp
> root triggers a sandbox denial on this machine."*

The `scratch` fixture is deliberately **not** cleaned up on teardown, and the docstring says why: *"the
sandbox's safe-delete guard rejects the recursive delete, and a test that writes into the repo's own
ignored scratch directory is harmless. Each test gets a unique path so no state leaks between them."*
This is the same guard that causes §7's four `test_safe_delete_shim` failures — documented here as a
design constraint rather than hidden as a quirk.

### 4.8 The public surface

`test_module_public_surface_is_declared` pins the exported names, and
`test_serialises_through_the_schema_serialiser` pins that the collection serialises through the *schema*
serialiser (not a bespoke one), so the wire form and the evidence form cannot drift. The shorthand and
helpers are pinned by `test_aggregate_evidence_shorthand_matches_the_engine`,
`test_collection_helpers_filter_correctly`, `test_collection_summary_reports_the_observable_facts`,
`test_digest_changes_when_a_claim_changes`, `test_digest_is_stable_across_regenerated_uuids`,
`test_collection_is_iterable_and_sized`.

`test_digest_is_stable_across_regenerated_uuids` is the one that makes §9 possible: because the digest
ignores process-local uuids, a recorded run's evidence digest can be recomputed in a *different process*
and still match. Without it, "recompute the verdict from the record" would be a claim you could not test.

### 4.9 What the evidence-engine suite does NOT establish

- It does **not** establish that the specialists produce correct evidence — only that the *aggregator*
  is deterministic and lossless given whatever they produce.
- It does **not** establish end-to-end reproducibility of a live run; that is §8's job.
- It does **not** establish that the calibration is *good* — the tests pin that calibration is
  *honestly labelled*, not that it improves anything (it does not; ECE got worse).

---

## 5. The frontend live-wiring suite

### 5.1 What it proves, and why it reads source text

`tests/unit/test_frontend_live_wiring.py` is the regression net for the frontend↔orchestrator boundary.
Its docstring states two things it proves, **both of which were broken before the change**:

> *"A. **CORS.** The deployment allowed exactly one origin (`https://satquery.pages.dev`), so the
> frontend could not be driven from a local development server at all: every request from
> `http://localhost:8080` was refused with `Disallowed CORS origin`, and a developer had to deploy to
> Cloudflare to test a one-line JavaScript change. The fix adds explicit localhost origins while keeping
> production listed and keeping `*` rejected.*
>
> *B. **The task vocabulary.** The page's router and the server's `Task` enum must agree. They are two
> independently written vocabularies in two languages, and nothing previously checked that a value the
> frontend would send is a value the server accepts. This module reads BOTH and compares them."*
> — `tests/unit/test_frontend_live_wiring.py`

And it pins a third, subtler defect — one invisible to a validity check:

> *"It also pins the `chang` word-boundary defect: `\bchang\b` cannot match 'changed', so the page's own
> default question ('What changed here?') routed to `vqa` instead of the change path. That is invisible
> to any test that only checks 'is this a valid enum value' — both branches produce a valid value. The
> test therefore asserts the ROUTE, not just the validity."*

That last sentence is the philosophical centre of the whole suite: **assert the route, not the
validity.** A test that asks "is `vqa` a valid task?" passes for both the correct and the defective
router, because both emit valid tasks. Only a test that asks "does *this question* route to *this
task*?" can see the defect.

**Why source-text parsing.** The module reads the frontend source rather than importing it, and says so:

> *"the frontend is plain ES5 with no build step and no module exports, so there is no importable
> symbol. Parsing the source is the only way to assert the shipped bytes. The parsing is anchored on
> named tokens rather than line numbers, so it does not silently pass if the file is restructured."*

The anchors are module-level path constants — `REPO_ROOT`, `DEPLOY_RENDER`, `FRONTEND_JS`, `MISSION_JS`,
`LIVE_JS`, `CORE_JS`, `MISSION_HTML` — and the production origin is a single constant,
`PRODUCTION_ORIGIN = "https://satquery.pages.dev"`. The CORS half imports the orchestrator with a clean
environment via `_render_module()`, because `_allowed_origins` reads `os.environ` at **call** time while
`_DEV_ORIGINS` is built at **import** time — a distinction the module's helper comment records
explicitly.

### 5.2 The measured result

**106 passed** (`release/CURRENT_RELEASE_STATE.md:121`; `README.md` §The test suites;
`docs/EVALUATION.md` §7.1). The command is:

```bash
.venv/Scripts/python.exe -m pytest tests/unit/test_frontend_live_wiring.py -q     # 106 passed
```

This is a **re-run-this-session** figure, and it is the number this chapter quotes.

### 5.3 The fifteen test classes

The classes were read from the file. Each name is a claim:

| # | Class | What it pins |
|---|---|---|
| 1 | `TestTheCorsAllowlistKeepsProductionAndAddsDevelopment` | production origin always allowed, not duplicated, dev origins added, `*` rejected even when hidden in a list |
| 2 | `TestTheCorsDecisionMatchesTheAllowlist` | the response-leg decision matches the allowlist (echo, no-headers, lookalike-host rejection) |
| 3 | `TestTheFrontendSpeaksTheServersTaskVocabulary` | every mapped value is a real server task; the mapping covers every task the router can emit; specialist names are human labels, not tasks |
| 4 | `TestTheDefaultQuestionReachesTheChangePath` | the page's own default question routes to the change path; the change stem matches its inflections; the temporal slot is required for a change question |
| 5 | `TestALocationQuestionIsNotAChangeQuestion` | a "where" question reads as grounding, with one or two assets |
| 6 | `TestTheArchitecturePolicyDoesNotReadALocationAsAChange` | the architecture sample routes to grounding; a place word in a "where" question is not a change |
| 7 | `TestTheTaskRespectsThePairRequirement` | paired tasks substitute correctly when given one asset; single-asset tasks are never substituted |
| 8 | `TestTheLiveClientUsesTheRealIngestionPath` | the client targets the orchestrator's routes; uploads send raw bytes with a derived content type; the infer body carries asset ids under `assets`; the client never sends a filesystem path |
| 9 | `TestThePageLoadsTheLiveClient` | `mission.html` includes live JS before mission JS; the declared API base is an absolute origin, not the relative fallback, not the frontend origin; no fixture image is wired into the analysis path |
| 10 | `TestTheLivePathKeepsTheHonestFailureContract` | a failure is attributed to the step that failed; the preview is kept for the no-file case |
| 11 | `TestTheCaptionStopsCallingARealUploadIllustrative` | a successful run rewrites the leading word and the alt text; the failure path does not claim an analysis happened; a new file clears the previous run's claim |
| 12 | `TestTheFrontendSendsOnlyTheAssetsTheTaskRequires` | a single-asset task with a pair uploaded uses only `t1`; the old wiring would have sent both; change/optical-SAR send the pair when present |
| 13 | `TestDescriptiveQueriesRouteToCaption` | descriptive queries reach caption; a description is not mistaken for change; caption is single-asset |
| 14 | `TestTheOpticalSarInputValidation` | a real optical+GeoTIFF SAR pair is ok; two plain photos warn; a missing SAR image is an error; the warning names the optical+radar expectation |
| 15 | `TestTheServerErrorTranslation` | invalid request gets asset guidance; 422 gets malformed guidance; recoverable `false` gets restart guidance; the server's message and detail are preserved; a null error is handled; a recoverable transport code is not called unrecoverable |

Two of these deserve a note because they encode *counter-intuitive* decisions:

**`TestTheFrontendSendsOnlyTheAssetsTheTaskRequires` (class 12).** The class contains
`test_the_old_wiring_would_have_sent_both_files_for_vqa` and
`test_the_fix_sends_one_file_where_the_old_sent_two`. This is a *differential* test: it asserts not just
that the new code is right, but that the *old* code was wrong in the specific way claimed. That is the
falsification discipline (§1.3) applied inside a suite — the fix is only meaningful if the defect it
fixes is real, and the test proves the defect by asserting the old behaviour differs.

**`TestTheCaptionStopsCallingARealUploadIllustrative` (class 11).** This is a *copy-honesty* test. The
page used to caption a real analysis "illustrative"; the suite asserts the leading word and the alt text
are rewritten on success, and — crucially — that the **failure path does not claim an analysis
happened** (`test_the_failure_path_does_not_claim_an_analysis_happened`). A UI that says "analysis
complete" on a failure is a claim defect, and it is tested as one.

### 5.4 The historical count: 94 → 100 → 106

A reviewer may see a different number. `docs/FINAL_DELIVERY_REPORT.md §7` records **94 passed** for this
file, while the current figure is **106**. The difference is the suite growing after the delivery report
was written: the regression tests added for the router fix took this file from **100 → 106** tests
(`DELIVERY_REPORT_2026-09-25.md`, regression-tests note). This chapter quotes **106** and cites both
figures so neither looks like a contradiction (`docs/REPRODUCIBILITY.md` §5.2).

### 5.5 What the frontend live-wiring suite does NOT establish

- It does **not** run the browser. It asserts the *shipped bytes* of the frontend against the
  *orchestrator module* in-process. Whether the page actually loads and runs is §8's job (live
  validation).
- It does **not** prove the router's *quality* — only that specific questions route to the specific
  tasks asserted. The router's residual misroutes (`"What is the new runway?"` → `change`) are real and
  recorded (`docs/LIMITATIONS.md`; `LIVE_VALIDATION_POSTFIX.md`), not hidden by a green suite.
- It does **not** test the inference service. It tests the boundary the frontend speaks to, which is the
  orchestrator.

---

## 6. The doc/frontend suite

### 6.1 The five files, and the measured result

The "doc/frontend suite" is a named grouping of **five files** that together report **183 passed**
(`docs/FINAL_DELIVERY_REPORT.md §7`; `docs/EVALUATION.md` §7.2; `docs/REPRODUCIBILITY.md` §5.3). The
five are:

| File | What it pins |
|---|---|
| `tests/unit/test_frontend_guide_doc.py` | both documents agree on tasks, coordinate systems, evidence types, geometry shapes; no login screen; no secrets in the browser; the confidence-property trap; serialized analyses |
| `tests/unit/test_frontend_live_wiring.py` | the frontend↔orchestrator boundary — see §5 |
| `tests/unit/test_api_contract_doc.py` | every JSON block parses and validates; no contract example fabricates an artifact URI; documented task/coordinate-system values are the real ones; every error code documented; the frozen config hash is recorded; the calibration caveat is recorded |
| `tests/unit/test_runbook_doc.py` | the runbook names only public serving entrypoints; quoted artifact sizes match the real files; the capability list matches the registry; env vars are the architecture ones; undecided items declared undecided; **no claim that a deployment happened**; verified distinguished from designed |
| `tests/unit/test_deploy_config.py` | the deploy manifest declares the non-registry marker; the validator reports no errors; the **config-hash regression guard**; the C-8 constraints (`torch_compile` false, cpu mode required, lazy load, single model cache) |

These are **documentation conformance tests**: they fail if a doc drifts from the code it describes.
They are the mechanism that keeps this release's documentation honest — the reason a doc claim can be
trusted is that a test would go red if the claim stopped matching the code.

### 6.2 Why four of the five are doc-guards, and one is not

Four of the five (`test_frontend_guide_doc`, `test_api_contract_doc`, `test_runbook_doc`,
`test_deploy_config`) are doc-guards, analysed individually in §3. The fifth
(`test_frontend_live_wiring`) is not a doc-guard — it is a code-behaviour suite that happens to be
grouped here because it reads frontend source text. The grouping is by *audience* (frontend +
documentation) rather than by mechanism, and that is why the count 183 covers both.

### 6.3 The doc-guard guarantee, restated

The doc-guards guarantee **example-level and value-level** conformance, not prose truth (§3.7). The two
module docstrings state the limit themselves:

> *"These tests validate the EXAMPLES, not the prose. They cannot detect a wrong sentence (e.g. a
> mis-stated latency budget)."* — `tests/unit/test_api_contract_doc.py`

> *"Prose can still be wrong in ways these tests cannot see -- a mis-stated latency, a wrong reason for a
> decision."* — `tests/unit/test_frontend_guide_doc.py`

`test_runbook_doc.py` reaches a different class of claim — numbers, names, and **status honesty** — and
its most important assertion is that the document itself *labels its unimplemented sections as
unimplemented* (§3.4). That is the guard against the exact overclaim the style guide forbids.

### 6.4 The config-hash regression guard

`test_deploy_config.py` carries the anti-drift move that makes it more than a validator: it does **not**
re-declare the frozen config hash. It imports it:

> *"We import it rather than re-declare it, so the two cannot drift."*
> — `tests/unit/test_deploy_config.py`

The authority is `tests.test_config`, and the value is the frozen hash `78f1e3700da15aa1`
(`DOCS_STYLE_GUIDE.md` §3). A second copy of the hash anywhere in the tree would be a latent
contradiction; the import makes drift structurally impossible.

The same file proves the manifest is **INERT** — *"it is never merged into the config registry and
cannot move `Config.hash`"* — and that negative cases use in-memory dicts via `validate_documents` *"so
the real config files are never mutated."* It also drives the validator CLI **in a child interpreter**
to prove the module's own `sys.path` bootstrap makes `core` importable *"WITHOUT pytest's `pythonpath =
.` shortcut"* (`test_validator_cli_exits_zero_as_a_standalone_script`).

### 6.5 What the doc/frontend suite does NOT establish

- It does **not** establish that any deployment exists. `test_runbook_doc.py` explicitly asserts the
  opposite direction: that the document does **not** claim a deployment happened (§3.4).
- It does **not** validate prose (§6.3).
- It does **not** check that the *code* is correct — only that the docs match the code. If the code and
  the docs are wrong together, a doc-guard is green.

---

## 7. The full `tests/unit` result, and the 5–6 environmental failures

This is the section where the release's honesty discipline is most visible. Running the entire unit tree
produces failures. They are reported, classified, and *not* reclassified as passes.

### 7.1 The result

Running `python -m pytest tests/unit` in the authoring sandbox produces **5–6 failures**. Every one is
**environmental or ordering**-related, not a regression in shipped code. The classification, exactly as
`docs/FINAL_DELIVERY_REPORT.md §7` records it:

| # | Failure | Attribution | Regression? |
|---|---|---|---|
| 1–4 | `test_safe_delete_shim` — **4 failures** | the sandbox's bulk-**delete guard** (Windows verbatim-path behaviour) | **No** |
| 5 | one **ordering flake** in the router route test | test **ordering** (passes in isolation) | **No** |
| 6 | one **stale adapter test** (`optical_sar` absent when CROMA unshipped) | stale test — **CROMA is now shipped** | **No** |

### 7.2 The evidence that they are not regressions

The evidence is the **re-run**. Re-running the affected files **together** gives **137 passed**
(`docs/FINAL_DELIVERY_REPORT.md §7`; `docs/EVALUATION.md` §7.3–7.4). The logic is stated plainly:

> *"if the failures were real regressions in the code under test, re-running those files together would
> still fail. They do not — which is what distinguishes an environmental/ordering failure from a
> regression."*

This is the falsification discipline (§1.3) applied to a *test result* rather than to a *guard*: the
claim "these are not regressions" is itself tested, by a re-run designed to fail if it were false.

### 7.3 The four `test_safe_delete_shim` failures, in detail

`tests/unit/test_safe_delete_shim.py` (338 lines, `pytestmark = pytest.mark.unit`) guards two defects in
the WorkBuddy Windows safe-delete shim (`cli/vendor/shim/sitecustomize.py`) that produced **phantom test
failures** in this repository. The module's docstring records both with measurements.

**Defect 1 — a verbatim temp path was not recognised as a temp path.**

> *"`_path_for_compare` returned `normcase(realpath(abspath(path)))`, which preserves the Windows verbatim
> prefix `\\?\`. `os.path.relpath` cannot relate `\\?\c:\...` to `c:\...`, so `_is_under_root` returned
> False and the OS-temp exemption in `_should_bypass_safe_delete` did not apply."*

Measured, before the fix:

```
plain    temp subdir  -> bypass True
verbatim temp subdir  -> bypass False      <-- the bug
non-temp subdir       -> bypass False
```

That is why routine pytest `garbage-*` collection — which walks
`\\?\C:\Users\...\Temp\pytest-of-anish\garbage-*` — reached the bulk guard, tripped `confirmRequired` at
**69 entries against a threshold of 50**, and **latched a rejection that then blocked every delete in
the conversation**.

**Defect 2 — a successful delete was reported as a failure.**

> *"`_platform_trash` raised whenever `SHFileOperationW` returned non-zero."*

Measured on the authoring host with `FOF_ALLOWUNDO|FOF_NOCONFIRMATION|FOF_NOERRORUI|FOF_SILENT`, calling
shell32 directly (the shim not involved):

```
series                                   non-temp rc      temp rc
4 regions x 3 interleaved rounds         2, 2, 2, ...     0, 0, 0
6 processes x 20 deletes                  2 (120/120)      0 (120/120)
```

The target is removed in **every** case and the user's Recycle Bin is populated and active (1,300+
`$I`/`$R` entries), so the return code is **not** a reliable "could not delete" signal. The module is
explicit that the code is **intermittent** and that the precise Windows-internal trigger is not known:

> *"across pytest invocations the same non-temp delete sometimes returned 0, and in one run a *temp*
> delete returned non-zero. The precise Windows-internal trigger is **NOT established** and is not
> claimed here."*

This is a place where this chapter writes **`UNKNOWN — not established from the available evidence`**,
following the module's own honesty.

**What the guard asserts** (the `WHAT IS ASSERTED` block of the docstring):

- A verbatim-prefixed path inside the OS temp root compares equal to its plain form and is exempted.
- A verbatim-prefixed path *outside* the temp root is still guarded — *"the fix normalises the prefix, it
  does not widen the exemption."*
- Deleting through a verbatim temp path succeeds end to end.
- A non-zero shell code on a delete whose target is genuinely gone does not raise (stubbed shell, stubbed
  `lexists` — deterministic, deletes nothing).
- A non-zero shell code on a delete whose target **survives** still raises — *"Fail-closed is preserved;
  only the false positive is gone."*
- Against the real shell, a non-temp delete does not raise (behavioural confirmation, not the guard).

**Why the guard is stub-based.** The module states the reason, and it is a direct consequence of defect
2's intermittency:

> *"the guard for this defect stubs the shell: a test that depends on the real return code would
> sometimes pass against the broken shim. The stub tests fail on the original shim on every run."*

That is the difference between a test that *sometimes* catches a defect and a test that *always* does.

**The rule for extending the file** is stated in the docstring and is worth reproducing, because it is
the exact failure this file exists to prevent:

> *"**Do not perform a real delete outside the OS temp root.** … An earlier draft of the defect-2 test
> did exactly that, tripped `confirmRequired` at the threshold and latched a rejection that blocked every
> subsequent delete in the session — the very failure this file exists to prevent."*

The module also **skips cleanly** when the shim is absent or disabled, via
`pytest.importorskip("sitecustomize", ...)` plus `skipif` markers (`needs_shim_helpers`, `windows_only`).
So on a normal user environment these tests skip; in the authoring sandbox they reach the guard and 4 of
the 8 fail. The constants involved are `VERBATIM = "\\\\?\\"` and a `NON_TEMP` probe path that is *"never
created"* — the predicates tested are pure.

### 7.4 The ordering flake

One router route test fails only when the full suite runs in a particular order, and **passes in
isolation**. That is the definition of an ordering flake: the failure is a function of collection order,
not of the code. `docs/REPRODUCIBILITY.md` §5.4 records it in the same row-set as the shim failures.

### 7.5 The stale adapter test

One test asserts `optical_sar` is **absent when CROMA is unshipped**. CROMA is now shipped
(`CURRENT_RELEASE_STATE.md` §1 capability contract: `optical_sar` `available: true`), so the assertion's
premise has changed. This is a **stale test**, not a code defect: the test encodes an assumption about
the environment that the environment has outgrown.

### 7.6 A different environment state: the 2,237-passed run

`docs/PHASE12_115_METRIC_COMPUTED.md` §8 records, for **2026-09-22**, a full unit suite result of
**2,237 passed, 0 failed (622.68 s)**. That is a real, measured result in a *different* environment state
(the safe-delete shim's guard state and the CROMA shipping status differ). It is recorded rather than
suppressed, *"because the two results are both true and the difference is exactly the
environmental/ordering story"* (`docs/EVALUATION.md` §7.3).

The same document records a correction that is itself an honesty lesson: an earlier revision claimed
`2,179 passed` computed as `2,171 + 8`, *"presented as a check but the total was never measured — it was
inferred from a stale baseline."* The measured figure was **2,237**. This is the style guide's rule
(`DOCS_STYLE_GUIDE.md` §1.3) in the wild: an inferred number presented as a measurement was corrected to
the measured one.

### 7.7 The UNKNOWN: the collected count

The exact **collected** test count for the current-session full-suite run is
**`UNKNOWN — not established from the available evidence`**. The record gives the failure classification
and the 137-passed re-run, but not a collected total. Per-suite counts are known (106, 183, 137, 73, 51);
the single collected total is not.

### 7.8 Why this is reported rather than hidden

The style guide's first rule is `DO NOT FABRICATE`, and its status rule is *"a mixed result is never 'all
work perfectly'"* (`DOCS_STYLE_GUIDE.md` §1.7). A full-suite run with 5–6 failures is a mixed result.
Presenting only the 106 and 183 would be exactly the "all work perfectly" overclaim the guide forbids.
The correct presentation is the three-row table (§7.1) plus the re-run evidence (§7.2) plus the explicit
UNKNOWN (§7.7) — which is what this section is.

---

## 8. The live-validation harness and its integrity discipline

### 8.1 What live validation is, and why it is separate from the suites

Every suite in §2–§7 runs in-process. `docs/API_CONTRACT.md` §8 states the boundary plainly: *"no test in
this repository dials a network address."* A green `tests/integration` run is therefore **not** evidence
about a deployment. **Live validation** is the separate activity that drives the *deployed* stack — the
real Cloudflare Pages frontend, the real Render orchestrator, the real tunnel, the real Codespace — with
a browser, and records what happened.

The record lives at `.workbuddy-ai/scratch/live_validation/` — `LIVE_VALIDATION_POSTFIX.md`, the raw
harness outputs (`run_output.txt`, `run_final2.txt`, `run_final3.txt`), the recomputed verdicts
(`results_final.json`, `results_pass3.json`), and eight per-case screenshots (`A1_vqa.png` … `B2_newairport.png`).

### 8.2 The three passes, summarised

| Pass | HEAD | Harness | Result |
|---|---|---|---|
| 1 | `ff46eba42b18` + `d413d3672311` | v1 (`fill_input`) | 8/8 |
| 2 | `2d7ae53b482d` | v2 asserting | 8/8 |
| 3 | `2d7ae53b482d` | v2 asserting (pre-discriminator-fix) | 8/8 (recomputed) |

**24 live runs, 24 correct dispatches, 0 mock nodes, no run id repeated across passes.** The trace fill
was **94.4444 %** on every case, and every case carries a real `run_*` id, the Hugging Face link in the
DOM, and the `capabilities → assets → infer` call sequence all addressed to
`<backend-host>` (two `assets` calls for the pair tasks).

The eight cases are Phase A (six regression cases: `vqa`, `caption`, `grounding`, `change`,
`change_vqa`, `optical_sar`) and Phase B (the two defect cases: `"Where are the built-up areas in this
image?"` and `"Where is the new airport?"`, both expected to reach `grounding` with a **single** asset —
the exact condition under which the old router collapsed to `vqa` and answered "River").

### 8.3 The CDP synthetic-key-event trap

This is the integrity lesson that shaped the harness, and it is the most transferable finding in this
chapter.

**The shape of the false pass.** Pass 1's harness drove the query box with `fill_input()`, which types
with **real CDP key events**. A re-run attempt failed on case 1 with `run_id=0002`, `mock_nodes=9`,
`answer="No answer yet"`, and only a `capabilities` call — the **mock path**. The root cause, measured
directly:

> *"`fill_input` types with **real CDP key events**, and Chrome **drops synthesized key events when the
> browser window does not hold OS focus**; it has **no assertion**, so it clicked Run with the page's
> default query still in the box. Measured directly: with Chrome backgrounded, `press_key("Z")` left
> `#qtext.value` unchanged, while `type_text("Q")` (CDP `Input.insertText`, not focus-gated) inserted
> fine."*
> — `LIVE_VALIDATION_POSTFIX.md`

The measurement is recorded in `diag_focus.harness`, which prints the value of `#qtext` after
`press_key("Z")` and after `type_text("Q")` — showing the first is a no-op and the second works.

**The generalisation.** A harness that types into a form and then reads a *result* cannot tell the
difference between:

- "the query was entered, and the run used it", and
- "the query was never entered, and the run used the page's default".

Both produce a result. The result of the second is a *plausible* answer to a *different* question, and a
harness without an assertion records it as the verdict for the first. That is a **silent false pass**,
and it is the reason the fix is not "use a different typing helper" but "**assert the form state before
dispatch**".

### 8.4 The fix: deterministic query entry plus pre-dispatch assertions

The harness was rebuilt as `run_all_postfix2.harness`, whose header states the change:

> *"v2 sets the query deterministically (js value + input event, falling back to `Input.insertText`) and
> ASSERTS the form state before clicking, recording `q_ok` / `obs_ok` / `t0_ok` and a computed verdict
> per case."*

The three assertions, and the failure each prevents:

| Assertion | Checks | Failure it prevents |
|---|---|---|
| `q_ok` | `#qtext.value` holds the intended query | the silent-drop false pass |
| `obs_ok` | `#obsTail == 'ready'` **and** one file on `#fileInput` | an upload that did not land |
| `t0_ok` | both frames present, for pair tasks | a pair task run on one asset |
| `no_mock_nodes` | `mock_nodes == 0` | the preview path being recorded as live |

The rule, stated in the session handoff (`HANDOFF_NEXT_AGENT.md` §5.2) and reproduced in
`docs/architecture/09-frontend.md` §10.3:

> *"**Do NOT use `fill_input()` or `press_key()` to enter the query.** They type with real CDP key events,
> which Chrome **silently drops when the browser window does not hold OS focus** … Use `js()` to set
> `#qtext.value` (plus `input`/`change` events) and/or `type_text()` (CDP `Input.insertText`, not
> focus-gated). **Always assert the form state before clicking Run** … or a no-op will be recorded as a
> pass."*

The `set_query()` function in the harness implements exactly this: it focuses the box, clears it, tries
`type_text(q)` (the non-focus-gated CDP path), reads the value back, and **falls back** to a direct
`js` set with `input`/`change` events if the read-back does not match. It returns the value actually read
back, and `q_ok = (qgot == c["q"])`. The assertion is on the *read-back*, not on the *attempt*.

`obs_ok`'s second half is a real DOM fact: `#obsTail` reads `none` in the markup (`mission.html:67`) and
the live driver flips it to `ready` on upload, so `obs_ok` proves the upload landed rather than merely
that a click happened.

### 8.5 The three discriminators that proved the earlier 8/8 was clean

The response to a silent-false-pass risk is not to re-run and hope; it is to check whether the *earlier*
result was contaminated, using evidence the harness recorded. The report does exactly that:

> *"I then checked whether the earlier 8/8 run was infected by the same silent failure. It was not:*
>
> * *its recorded intents are **query-specific** — A1 reads `taskvqa…temporalnone`, whereas the default
>   query "What changed here?" would read `taskchange…temporalrequired` (exactly what the failed run
>   showed);*
> * *its answers **embed the query text** — e.g. `[grounding] Located 6 candidate region(s) for 'Where
>   are the built-up areas in this image?'`;*
> * *A6 required two files (`optical 4/12 + SAR 2/2` channels), which only the uploaded pair supplies.*
>
> *So the 8/8 result is a valid measurement."*
> — `DELIVERY_REPORT_2026-09-25.md` §3

The three discriminators generalise into a reusable checklist:

| Discriminator | What it proves |
|---|---|
| the recorded **intent** is query-specific | the query reached the router |
| the **answer** embeds the query text | the server received the intended query |
| a case **requires an artefact** only the setup supplies | the setup really happened |

This is why the harness records the full per-case payload — `intent`, `answer`, `files_t1`, `files_t0`,
`obs_tail`, `mock_nodes`, `api_calls` — rather than just a pass/fail bit. The bit can be wrong; the raw
record can be re-examined.

### 8.6 The two discriminator bugs (and why they were false *failures*, not false passes)

Two harness bugs were found and fixed, both causing **false failures** — the safe direction:

1. **The answer `[task]` tag exists only for region tasks.** `vqa`/`caption` answers are bare
   ("Grassland", prose), so a tag-based verdict heuristic reported them as failures. The fix computes the
   dispatched task as `answer_tag` when present, else the intent's reading.
2. **The intent panel is a concatenated string.** `task([a-z_]+)` must be matched **non-greedily** up to
   `modality`; a greedy match swallows the whole string. The fix is
   `re.search(r"task([a-z_]+?)modality", intent)`.

The harness header states the asymmetry: *"Two further harness bugs were found and fixed, both causing
**false failures**."* A false failure costs a re-run; a false pass costs the integrity of the result. The
harness was built so that its bugs err toward false failure, and §9 makes even the false-failure case
recoverable without a re-run.

### 8.7 Liveness traps discovered during the runs

Two operational traps are recorded because they look exactly like stalls:

- **`browser-use` block-buffers stdout even when redirected to a file**, so the output file stays at
  **0 bytes until the process exits** — indistinguishable from a stall. The harness now forces
  line-buffering (`sys.stdout.reconfigure(line_buffering=True)`) and prints `CASE_START <id>` per case,
  so the file grows case by case.
- **`grep` block-buffers when piped**, so piping the harness through `grep` swallows all output if the
  pipeline is killed. Redirect to a file instead.

The harness also records `ALL_DONE` as a final marker, so a truncated run is detectable from the output
file alone.

### 8.8 What the live validation does NOT establish

- It is **not** the independent audit. `LIVE_VALIDATION_POSTFIX.md` opens by saying so: *"This is a
  *re-validation*, not the independent audit: the prior independent audit is
  `../2026-09-25-00-37-41/live_validation/INDEPENDENT_LIVE_VALIDATION.md`."*
- It does **not** establish model *quality*. The two verdicts are kept separate: **Deployment/integration:
  PASS** and **Model quality: MIXED** (caption and grounding meaningful; change/change_vqa plausible; VQA
  weak-but-related — A1 answers "Grassland"; optical-SAR returns a bare class index `class_18`, not a
  human label).
- It does **not** establish load behaviour, latency ceilings, or cold-start times. It is eight cases, run
  three times.
- It does **not** clear B-07: transient tunnel-agent gaps remain **OPEN**, and the patch is **prepared,
  NOT deployed** (`CURRENT_RELEASE_STATE.md` §6).

---

## 9. Verdict-independent evidence: recomputing from raw records

### 9.1 The property

The live-validation harness has a `verdict` field. The project's design decision is that **the verdict
must not be the primary record.** The recorded evidence — `q_ok`, `obs_ok`, `t0_ok`, `run_id`,
`mock_nodes`, `intent`, `answer`, `api_calls` — is recorded *independently of* the verdict computation,
so a harness bug in the verdict logic can never silently turn a real failure into a pass, and can be
corrected without a re-run.

### 9.2 The recomputation script

`.workbuddy-ai/scratch/recompute_verdicts.py` implements the property. Its docstring states the case that
motivated it:

> *"The v2 harness recorded, per case: `q_ok`, `obs_ok`, `t0_ok`, `run_id`, `mock_nodes`, `intent`,
> `answer`, `api_calls` -- all independent of the verdict computation. Its verdict field was wrong for
> vqa/caption because the tag heuristic assumed every answer starts with `"[task]"`; vqa/caption answers
> are bare. This recomputes the verdict from the intent's `task<name>` slot (primary) plus the answer tag
> (cross-check only when present)."*

The script is read-only over the recorded run and writes `results_final.json`. The core of it:

```python
mi = re.search(r"task([a-z_]+?)modality", intent)
intent_task = mi.group(1) if mi else ""
tag = answer[1:answer.find("]")] if (answer.startswith("[") and "]" in answer) else ""
expect = d["expect"]

# The DISPATCHED task is what the server actually ran. The answer carries a "[task]"
# prefix for the region tasks (grounding/change/change_vqa/optical_sar); vqa and
# caption answers are bare, so fall back to the intent's reading there.
# NOTE: A5 is a legitimate case where the two differ -- the router READS "change" but
# dispatches "change_vqa" (the documented quantifier upgrade). That is not a failure.
dispatched = tag if tag else intent_task

checks = {
    "query_in_box": bool(d.get("q_ok")),
    "asset_attached": bool(d.get("obs_ok")),
    "pair_attached": bool(d.get("t0_ok")),
    "live_run": str(d.get("run_id", "")).startswith("run_"),
    "no_mock_nodes": d.get("mock_nodes") == 0,
    "task_dispatched": dispatched == expect,
}
verdict = "PASS" if all(checks.values()) else "FAIL"
```

Every input to the verdict is a *recorded observation*; the verdict is a pure function of them. The
script also records `read_vs_dispatched_differs` — so the one legitimate read≠dispatch case (A5, the
documented quantifier upgrade) is **flagged, not failed**.

### 9.3 Pass 3 is the proof of the property

Pass 3 was launched with the harness build that still carried the two discriminator bugs (§8.6), so its
raw output says **`SUMMARY 0/8`**. The verdicts were recomputed from the recorded evidence by
`recompute_verdicts.py` and are **8/8 PASS** (`results_pass3.json`). The record states the significance:

> *"That is the intended safety property — **the recorded evidence (run id, `mock_nodes`, intent, answer,
> assertions) is independent of the verdict computation**, so a harness bug can never silently turn a
> real failure into a pass, and never costs a re-run to correct."*
> — `LIVE_VALIDATION_POSTFIX.md`

This is the strongest form of the claim: the property was not merely designed, it was **exercised**. A
run with a broken verdict computation was corrected *post hoc* from its own raw record, without touching
the deployment or re-running the browser.

### 9.4 The generalisation

The pattern is: **record observations, compute verdicts separately, keep the observations.** It is the
same shape as the evidence engine's digest (§4.8): the *record* is content-addressed and process-independent,
and the *interpretation* is a function over it. Both make the same guarantee — that a conclusion can be
re-derived from the evidence rather than trusted as an assertion.

---

## 10. How to run the tests

### 10.1 The interpreter

Use the repository virtual environment, not a bare `python`. On Windows:

```bash
.venv/Scripts/python.exe -m pytest <target> -q
```

`docs/REPRODUCIBILITY.md` §10.2 records this as the invocation that reproduces the published results. On
Windows, pytest must be given `-p no:cacheprovider` because the sandbox refuses `.pytest_cache` writes
(`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §0).

### 10.2 Run targeted files, not the whole tree

**A full `tests/unit` run trips the sandbox's bulk-delete guard** (the 4× `test_safe_delete_shim`
failures of §7.3). The workaround is to run targeted suites:

```bash
.venv/Scripts/python.exe -m pytest tests/unit/test_frontend_live_wiring.py -q     # 106 passed
.venv/Scripts/python.exe -m pytest tests/unit/test_evidence_engine.py --basetemp=.pytest_tmp -q
.venv/Scripts/python.exe -m pytest tests/unit/test_deploy_config.py -p no:cacheprovider -q
.venv/Scripts/python.exe -m pytest tests/unit/test_api_contract_doc.py tests/unit/test_frontend_guide_doc.py -p no:cacheprovider -q
```

The runbook notes that **multi-suite invocations in one command have been refused by the environment
before; single suites are reliable** (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §2.6). The `--basetemp`
flag is **mandatory** for the evidence-engine suite (§4.7): the default pytest temp root triggers a
sandbox denial on this machine, and the module's `scratch` fixture writes to a repo-local `.pytest_tmp`
instead.

### 10.3 `pytest.ini`

The configuration is four lines plus the marker registrations:

```ini
[pytest]
testpaths = tests
pythonpath = .
addopts = -q --tb=short
```

(`pytest.ini:1-4`). `pythonpath = .` is what makes `from core.config import ...` resolve from the repo
root without an installed package — and `test_deploy_config.py` deliberately proves the module *also*
bootstraps `sys.path` itself, *"WITHOUT pytest's `pythonpath = .` shortcut"* (§6.4). The markers are
registered at `pytest.ini:23-27` (§0.1), and `filterwarnings` silences three warning classes including
`rasterio.errors.NotGeoreferencedWarning` (`pytest.ini:5-8`).

### 10.4 The output-buffering trap

**Do not pipe pytest through `grep`** in the authoring sandbox — output is block-buffered and a killed
pipeline swallows it. Redirect to a file instead (`docs/REPRODUCIBILITY.md` §5.6). The same trap applies
to the live-validation harness (§8.7).

### 10.5 The expected results

| Suite | Command | Expected | Status |
|---|---|---|---|
| Frontend live-wiring | `pytest tests/unit/test_frontend_live_wiring.py -q` | **106 passed** | `VERIFIED` |
| Doc/frontend suite | `pytest` on the 5 doc/frontend files | **183 passed** | `VERIFIED` |
| Evidence engine | `pytest tests/unit/test_evidence_engine.py --basetemp=.pytest_tmp` | **73 passed** | `VERIFIED` |
| Gateway policy | `pytest tests/unit/test_gateway_policy.py` | **51 passed** | `VERIFIED` |
| Full unit suite | `pytest tests/unit` | **5–6 environmental/ordering failures**, rest pass; a re-run passes **137** | `MEASURED` |

(`release/repo/README.md` §The test suites; `docs/PHASE19_FINAL_HARDENING.md` §6;
`docs/REPRODUCIBILITY.md` §5.1.) The precise full-suite **collected** count is
`UNKNOWN — not established from the available evidence` (§7.7).

---

## 11. What is NOT tested

A green suite is not evidence about a property nobody wrote an assertion for (`DOCS_STYLE_GUIDE.md` §1.6;
§0.3). This section lists the properties the release does **not** have test evidence for, so that no
reader mistakes coverage for completeness.

| Property | State | Why / what exists instead |
|---|---|---|
| **Continuous integration** | **does not exist** | there is **no `.github/` directory** and no CI workflow in the repository. Every test result in this chapter was produced by a **manual** run in the authoring sandbox. Nothing runs the suites automatically on push. |
| **End-to-end system benchmark** | **NOT RUN (none exists)** | `CURRENT_RELEASE_STATE.md` §4: *"System-level end-to-end benchmark — **NOT RUN (none exists)**."* No system-level accuracy is claimed anywhere in the release. |
| **Load / soak / concurrency** | **NOT RUN** | no test exercises sustained load, concurrent users, or a long-running process. The per-IP rate limiter is a **fairness** control, not a security or capacity control (`docs/DEPLOYMENT_ARCHITECTURE.md` §5.2). |
| **Adversarial / fuzz testing** | **NOT RUN** | no fuzzing or adversarial-input suite. Input validation is tested by example (§2.2.9), not by search. |
| **Cross-dataset generalization** | **NOT RUN** | the measured metrics are per-dataset (LEVIR-CD-256, VRSBench, held-out test sets); no test measures transfer to an unseen dataset. |
| **Browser / device matrix** | **NOT RUN** | live validation ran in **headed Chromium** only. No Firefox/Safari/WebKit, no mobile, no accessibility audit. |
| **Deployment verification in CI** | **NOT RUN** | the runbook guard asserts the document does **not** claim a deployment happened (§3.4). Deployment reachability is established only by the manual live validation (§8). |
| **Router test split** | **NOT RUN** | `CURRENT_RELEASE_STATE.md` §4: router accuracy `0.965116` is **validation, ungated, n = 86**; *"the test split was NOT RUN"* (`DOCS_STYLE_GUIDE.md` §3). |
| **Calibration as an improvement** | **not established (and measured the other way)** | ECE went `0.013755 → 0.014929` — **worse**. The engine suite pins that calibration is *honestly labelled* (§4.5), not that it helps. |
| **Optical-SAR / change-VQA acceptance** | **OPEN** | the rulings are OPEN (`CURRENT_RELEASE_STATE.md` §4; `DOCS_STYLE_GUIDE.md` §3). A suite passing is not a model acceptance. |
| **VLM adapter acceptance** | **REJECTED** | metrics `USABLE_VERIFIED` (exact_match 0.963) but status **ACCEPTANCE-REJECTED**. USABLE ≠ ACCEPTED (`DOCS_STYLE_GUIDE.md` §3). |
| **Tunnel reliability (B-07)** | **OPEN** | transient tunnel-agent gaps; patch **prepared, NOT deployed**. No test can cover a defect that is not fixed in production. |
| **The inference service's internals** | **UNKNOWN** | the tunnel-agent source is not in the monorepo; `docs/architecture/02-deployment-topology.md` §3.3/§9.4 documents the topology as a **measured** fact, but the agent's internals are `UNKNOWN — not established from the available evidence`. |
| **Credential write permission (HF)** | **UNKNOWN** | `CURRENT_RELEASE_STATE.md` §7: HF token identity `thundercode`, role `fineGrained`; *"write permission not yet proven."* |

Two of these deserve emphasis:

- **No CI is the largest gap.** Every count in this chapter is a manual measurement. A reader should not
  read "106 passed" as "the suite passes on every commit" — it means "it passed when it was run, in the
  recorded environment." There is no automation that would catch a regression on push.
- **No end-to-end benchmark is the second.** The individual task metrics are artifact-backed and
  measured, but there is no measurement of the *system* — router + specialist + envelope + frontend — on
  a held-out end-to-end corpus. The live validation (§8) proves the *pipeline runs and dispatches
  correctly*; it does not measure system accuracy.

---

## 12. Open items and evidence index

### 12.1 Open / blocked / not-run items for this chapter's topic

Following the style guide's requirement that every doc end with an explicit list
(`DOCS_STYLE_GUIDE.md` §4):

| Item | State | Note |
|---|---|---|
| B-07 — transient tunnel-agent gaps | **OPEN** | patch **prepared, NOT deployed**; root shape measured (in `auto` mode a tunnel timeout falls through to the forward path, burning `wake_timeout_s=120` on a 302 ≈ 249 s) |
| B-02 — `codespace_name` trailing `\n` | **OPEN (cosmetic)** | confirmed still live; the wake path strips it |
| No CI | **does not exist** | no `.github/` directory; all results are manual |
| System-level end-to-end benchmark | **NOT RUN (none exists)** | no system-level accuracy claimed |
| Router test split | **NOT RUN** | val-only, n = 86 |
| Full-suite collected test count | **UNKNOWN** | per-suite counts known; the single collected total is not |
| `test_safe_delete_shim` failures | **environmental** | sandbox delete-guard; not regressions; precise Windows trigger **UNKNOWN** |
| Ordering flake | **environmental** | passes in isolation |
| Stale adapter test | **stale** | asserts CROMA unshipped; CROMA is shipped |
| Live validation | **MEASURED** | 3 passes × 8 cases, 8/8 each, 24 runs, 0 mock nodes, trace fill 94.4444 % |
| Optical-SAR / change-VQA rulings | **OPEN** | 0.931 / macro-F1 0.434161; 0.697626 / 0.378373 |
| VLM adapter acceptance | **REJECTED** | metrics usable; acceptance rejected |
| Calibration | **not an improvement** | ECE worse (0.013755 → 0.014929); retained only because it is in the frozen config |

### 12.2 Where the evidence lives

| What | Where |
|---|---|
| Test configuration and markers | `pytest.ini` |
| Frontend live-wiring suite | `tests/unit/test_frontend_live_wiring.py` — **106 passed** |
| Evidence-engine suite | `tests/unit/test_evidence_engine.py` — **73 passed** |
| Safe-delete shim guard | `tests/unit/test_safe_delete_shim.py` — 338 lines; skips cleanly off-sandbox |
| Doc-guards | `tests/unit/test_api_contract_doc.py`, `test_frontend_guide_doc.py`, `test_runbook_doc.py`, `test_deploy_config.py`, `test_step7_api_contract_conformance.py` |
| Integration suite scope | `tests/integration/test_step7_backend_chain.py`, `test_step7_r02_serving.py`; `docs/ITEM5_INTEGRATION_SUITE_SCOPE.md` |
| The suite results, reported in full | `docs/EVALUATION.md` §7.1–7.5; `docs/REPRODUCIBILITY.md` §5; `release/repo/README.md` §The test suites |
| The full-suite failure classification | `docs/FINAL_DELIVERY_REPORT.md` §7; `docs/PHASE12_115_METRIC_COMPUTED.md` §8 (the 2,237-passed run) |
| Live-validation record | `.workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md` |
| Live-validation raw output | `.workbuddy-ai/scratch/live_validation/run_output.txt`, `run_final2.txt`, `run_final3.txt` |
| Live-validation screenshots | `.workbuddy-ai/scratch/live_validation/A1_vqa.png` … `B2_newairport.png` (8) |
| Live-validation harness (v1) | `.workbuddy-ai/scratch/run_all_postfix.harness` |
| Live-validation harness (v2, asserting) | `.workbuddy-ai/scratch/run_all_postfix2.harness` |
| CDP focus diagnostic | `.workbuddy-ai/scratch/diag_focus.harness` |
| Verdict recomputation | `.workbuddy-ai/scratch/recompute_verdicts.py` |
| Prior independent audit | `../2026-09-25-00-37-41/live_validation/INDEPENDENT_LIVE_VALIDATION.md` |
| Release-state inventory | `release/CURRENT_RELEASE_STATE.md` |

### 12.3 Cross-references

- The trust model, secret custody, CORS allowlist, size limits and rate-limiting-as-fairness that the
  frontend suite asserts against: [`SECURITY.md`](SECURITY.md).
- The API contract the doc-guards validate: [`API_CONTRACT.md`](architecture/08-api-contract.md).
- The deployment topology the live validation drives: [`DEPLOYMENT.md`](DEPLOYMENT.md),
  [`architecture/02-deployment-topology.md`](architecture/02-deployment-topology.md).
- The measured metrics this chapter does **not** re-derive: [`EVALUATION.md`](EVALUATION.md).
- The reproduction commands and their expected outputs: [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) §5.
- The router defect the live validation re-validates: [`architecture/04-router.md`](architecture/04-router.md),
  [`architecture/09-frontend.md`](architecture/09-frontend.md) §10.