release: add docs/LIMITATIONS.md

docs/LIMITATIONS.md

@@ -0,0 +1,82 @@
+# Limitations
+
+An honest catalogue of everything this system does **not** do, does poorly, or does not know. Negative
+results and open items are listed here rather than omitted.
+
+**Status tags:** `OPEN` · `NOT RUN` · `REJECTED` · `DEFERRED` · `BY DESIGN`.
+
+---
+
+## 1. Model quality
+
+| # | Limitation | Detail |
+|---|---|---|
+| 1 | **Grounding IoU is low in absolute terms** | mean best IoU 0.2838 (canonical) / 0.2566 (matched6). The trained head clearly beats the zero-shot baseline (0.0972), but 0.28 is not "solved". |
+| 2 | **Grounding depends on the box convention** | Two protocols and two decode variants produce very different numbers (0.2838 vs 0.1215 argmax). Absolute values are protocol-sensitive. |
+| 3 | **Optical-SAR accuracy is carried by common classes** | accuracy 0.931 but macro-F1 0.434161. 5 of 19 classes are absent in the scored split and contribute 0.0 to macro-F1 by construction. |
+| 4 | **Change-VQA is weak on rare classes** | accuracy 0.697626 / macro-F1 0.378373 (and 0.651469 / 0.372309 on the second test set). |
+| 5 | **VQA is weak-but-related** | the live VQA path answers broadly related content (e.g. "Grassland") rather than a crisp class. |
+| 6 | **Optical-SAR returns a bare class index** | the live service returns `class_18`, not a human-readable label. |
+| 7 | **Calibration made things worse** | ECE 0.013755 → 0.014929. Retained only because it is part of the frozen config. |
+| 8 | **The VLM adapter is not accepted** | metrics usable (exact_match 0.963), but the artifact is acceptance-rejected; the deployed path uses the unadapted model. |
+| 9 | **The router number is not a test result** | 0.965116 is validation, **ungated**, n = 86. The test split was **NOT RUN**. |
+
+## 2. Router residuals (known misroutes)
+
+| # | Query | Behaviour | Note |
+|---|---|---|---|
+| 10 | *"What is the new runway?"* | reads `change`, not `vqa` | residual ambiguity — "new" triggers a change reading |
+| 11 | *"How much built-up area was added?"* | reads `vqa` (under-trigger) | a change-style quantifier not caught by the router |
+| 12 | *"What changed between the earlier and later image?"* with one asset | console **reads** `change`, **dispatches** `change_vqa` | intentional (asset-count-aware dispatch) but visually surprising — documented in [`ARCHITECTURE.md`](ARCHITECTURE.md) §4 |
+
+## 3. Evaluation gaps
+
+| # | Gap | State |
+|---|---|---|
+| 13 | **No system-level end-to-end benchmark** | **NOT RUN — none exists.** No end-to-end accuracy is claimed. |
+| 14 | **Router test split** | **NOT RUN** |
+| 15 | **Benchmark adapters** | **NOT RUN** |
+| 16 | **End-to-end latency benchmark** | **NOT RUN** (per-specialist latency only, incidentally) |
+| 17 | **Cross-dataset generalisation** | **NOT RUN** — each specialist is evaluated on its own training-family split only |
+| 18 | **Human evaluation** | **NOT RUN** |
+| 19 | **Robustness / adversarial evaluation** | **NOT RUN** |
+| 20 | **BigEarthNet label semantics** | the local subset is 100 % single-label vs the official 1–11 multi-label scheme, so metrics are **not comparable** to published numbers |
+| 21 | **Reliability diagram** | the pre-scaling diagram is labelled as such; the calibrated curve is not plotted |
+
+## 4. Operational limitations
+
+| # | Limitation | State |
+|---|---|---|
+| 22 | **B-07: transient tunnel gaps** | a request can hang or return 504. Patch prepared, **NOT deployed**. **OPEN** |
+| 23 | **B-07 root shape** | in `auto` mode a tunnel timeout falls through to the forward path, burning `wake_timeout_s` (120 s) on a `302`; worst case ≈ 249 s. Measured. |
+| 24 | **Cold start is tens of seconds** | Render free tier sleeps; the Codespace may be stopped. Documented, not hidden. |
+| 25 | **B-02: `codespace_name` trailing newline** | cosmetic; the wake path strips it. **OPEN (cosmetic)** |
+| 26 | **No database, auth, or queue** | stateless gateway by design — no persistence of runs or users. |
+| 27 | **Deployment repos are private** | their links 404 for an outside audience. **BY DESIGN** |
+| 28 | **`deploy/` in the monorepo is stale/untracked** | not the deployed source. Trap. |
+
+## 5. Packaging and licensing
+
+| # | Limitation | State |
+|---|---|---|
+| 29 | **No `LICENSE` file** | none exists in the source repository. **OPEN** — a licence must be selected before public release of the code. |
+| 30 | **Backbones are not redistributed** | fetched from the Hub at run time; their licences are their own. |
+| 31 | **Artifact bloat** | `artifacts/` is ~3.7 GB, mostly reproducible caches, duplicate ZIPs and superseded checkpoints rather than released weights. |
+
+## 6. Documentation caveats
+
+| # | Caveat | State |
+|---|---|---|
+| 32 | **`docs/FINAL_DELIVERY_REPORT.md` §6 is stale** | it still lists the bundled EO change pair as DEGRADED and B-01 as BLOCKED; both were resolved on 2026-09-25. |
+| 33 | **The monorepo `README.md` was materially stale** | it described a hermetic frontend, an in-progress Render/Codespace, and a `/v1/*` contract. Superseded by this release's README. |
+| 34 | **`hf/` docs were stale** | they asserted the project owns no weights and has no HF credentials. Both were false at release time. |
+| 35 | **Anatomy plate image variant** | points at the 720×720 variant of an image the recorded run analysed at 730×730. Cosmetic. |
+
+## 7. Explicit non-claims
+
+- **No claim of state-of-the-art performance** on any benchmark.
+- **No claim of production readiness** for the model quality — the deployment runs, but the models
+  carry the limitations above.
+- **No claim that the trained heads generalise** beyond their training-family test splits.
+- **No claim that calibration improves confidence.**
+- **No claim that the VLM adapter is accepted** for production use.