release: add docs/DEPLOYMENT.md

docs/DEPLOYMENT.md

@@ -1,126 +1,325 @@
 # Deployment
 
-**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `OPEN` · `DEFERRED` · `BLOCKED`.
+**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `OPEN` · `DEFERRED` · `BLOCKED` · `BY DESIGN`.
 
-The live system is deployed across three **private** repositories plus one **public** umbrella repo.
-The monorepo working copy is **not** the deployed source. This document records the live topology,
-the environment variables, the deploy mechanics, and the traps that cost real debugging time.
+The live SatQuery AI system runs across **three private repositories** plus one **public umbrella
+repository**. The monorepo working copy is **not** the deployed source. This document records the live
+topology, every environment variable, the deploy mechanics, the cold-start behaviour, the known
+issues, and the platform traps that cost real debugging time.
+
+> **The single most important trap in this document.** `deploy/` inside the monorepo is **stale and
+> untracked**. It is **not** the deployed source. Edits must go to the three real repositories.
 
 ---
 
 ## 1. Live revisions (VERIFIED)
 
+Read from the GitHub API during the release reconnaissance.
+
 | Component | Repository | Visibility | Branch | Revision | Host |
 |---|---|---|---|---|---|
-| Frontend | `Anish-lab-blip/SatQuery-Frontend` | private | `main` | **`2d7ae53b482d`** | Cloudflare Pages → `satquery.pages.dev` |
-| Backend / orchestrator | `Anish-lab-blip/SatQuery-Backend` | private | `main` | **`89d80eaddec5`** | Render → `satquery-backend-m4yv.onrender.com` |
-| Inference | `Anish-lab-blip/SatQuery-Inference` | private | `main` | **`5a0936ace491`** | Codespace `potential-space-trout-r4ppw969w45j2pvvw`, port 8000 |
-| Public umbrella | `Anish-lab-blip/SatQuery-AI` | **public** | `main` | `3dcabd32da41` | this release home |
-| Monorepo (working copy) | `C:/Users/anish/satquery-ai` | local only | `master` | `9d57aed` | **no remote**; 334 dirty entries |
+| Frontend | `Anish-lab-blip/SatQuery-Frontend` | **private** | `main` | **`2d7ae53b482d`** | Cloudflare Pages → `satquery.pages.dev` |
+| Backend / orchestrator | `Anish-lab-blip/SatQuery-Backend` | **private** | `main` | **`89d80eaddec5`** | Render → `satquery-backend-m4yv.onrender.com` |
+| Inference | `Anish-lab-blip/SatQuery-Inference` | **private** | `main` | **`5a0936ace491`** | Codespace `potential-space-trout-r4ppw969w45j2pvvw`, port 8000 |
+| Public umbrella | `Anish-lab-blip/SatQuery-AI` | **public** | `main` | `3dcabd32da41` ("Initial commit") | this release home |
+| Monorepo (working copy) | `C:/Users/anish/satquery-ai` | local only | `master` | `9d57aed` | **no git remote**; 334 dirty entries |
 
-> **Trap.** `deploy/` inside the monorepo is **stale and untracked**. It is **not** the deployed
-> source. Edits must go to the three real repositories.
+The public umbrella repository previously contained **only a 13-byte README** (`# SatQuery-AI`). It is
+the intended home for this release.
 
-## 2. Live health (VERIFIED, probed)
+## 2. The four tiers
 
-```json
-{"status":"ok","service":"satquery-orchestrator",
- "tunnel":{"agent_connected":true,"agent_id":"codespaces-fd1038","pending":0,"completed":97},
- "config":{"codespace_name":"potential-space-trout-r4ppw969w45j2pvvw\n","codespace_port":8000,
-           "transport_mode":"auto","tunnel_timeout_s":150.0,"wake_timeout_s":120.0,
-           "upstream_timeout_s":90.0,"device":"cpu","has_github_token":true}}
 ```
+Browser
+  │  HTTPS
+  ▼
+Cloudflare Pages  —  satquery.pages.dev                (static frontend, 11 pages)
+  │  HTTPS / JSON  →  /api/*
+  ▼
+Render            —  satquery-backend-m4yv.onrender.com (orchestrator / gateway)
+  │  outbound long-poll  POST /tunnel/agent
+  ▼
+GitHub Codespace  —  FastAPI inference, CPU, port 8000
+  │  build_space_app()
+  ▼
+specialists: SmolVLM · RemoteCLIP · MiniLM · CROMA · STANet
+  │
+  ▼
+ResultEnvelope  →  tunnel  →  Render  →  browser
+```
+
+```mermaid
+flowchart LR
+  U[Browser] -->|HTTPS| CF["Cloudflare Pages<br/>static frontend"]
+  CF -->|"HTTPS JSON<br/>/api/health · /api/capabilities · /api/infer · /api/assets"| R["Render<br/>orchestrator / gateway"]
+  R -->|"outbound long-poll<br/>POST /tunnel/agent"| C["GitHub Codespace<br/>FastAPI inference :8000"]
+  C --> S[(SmolVLM · RemoteCLIP<br/>MiniLM · CROMA · STANet)]
+  C -->|ResultEnvelope| R
+  R -->|"envelope + error translation"| CF
+```
+
+## 3. Why each tier exists
+
+### 3.1 Cloudflare Pages — the static tier
 
-Note `codespace_name` still carries a trailing `\n` — the cosmetic B-02 patch is **not** deployed.
+Serves the frontend. **No backend, no secrets, and no API calls of its own** on the static pages. The
+one exception is the Analyze console, which calls the gateway. Staged by `scripts/stage_pages.mjs` and
+deployed with `npx wrangler pages deploy`.
 
-## 3. Environment variables
+### 3.2 Render — the gateway
 
-### 3.1 Render (gateway) — measured live
+A deliberately **thin, stateless** orchestrator. Its responsibilities, from
+`docs/DEPLOYMENT_ARCHITECTURE.md` §2:
 
-| Variable | Value (live) |
+| Responsibility | Detail |
 |---|---|
-| `CODESPACE_NAME` | `potential-space-trout-r4ppw969w45j2pvvw` |
-| `CODESPACE_PORT` | `8000` |
-| `SATQUERY_ALLOWED_ORIGINS` | `https://satquery.pages.dev` |
-| `SATQUERY_DEVICE` | `cpu` |
-| `SATQUERY_TRANSPORT` | `auto` |
-| `SATQUERY_TUNNEL_TIMEOUT_S` | `150` |
-| `SATQUERY_WAKE_TIMEOUT_S` | `120` |
-| `SATQUERY_UPSTREAM_TIMEOUT_S` | `90` |
-| `GITHUB_TOKEN` | present |
+| Schema validation | reject malformed requests before they cost inference |
+| Size limits | whole-request body cap, shared with the Codespace |
+| Rate limiting | per-IP count + window — **fairness, NOT a security control** |
+| CORS allowlist | the Pages origin; **never `*`** |
+| Request ids | correlate a request across tiers |
+| Timeouts | sit inside the task budget |
+| Secret custody | holds credentials that must never reach the browser |
+| Error translation | upstream failures → the documented error envelope |
+
+It is **not** a model host. It has **no database, no auth, and no queue**.
+
+> **Hard rule.** The gateway must **not** retry `POST /api/infer` on its own — a retry would consume
+> inference a second time. The client decides on retry.
+
+### 3.3 GitHub Codespace — the inference tier
+
+Runs the real inference service: `build_space_app()` from `app/space_app.py`, served by
+`deploy/codespace/serve.py` on `$PORT`, in **CPU mode**. It honours the four-endpoint contract
+(`/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets`), imports cheaply without torch, reuses
+`app/serving.py` as the composition root, and **degrades rather than crashes** on absent artifacts.
+
+### 3.4 Hugging Face — the model tier
+
+Holds the six trained artifacts and the model card. It is **not** the runtime inference host; the
+Codespace resolves the pinned backbones from the Hub at run time.
+
+## 4. Why the transport is an outbound tunnel
+
+The inference host is a Codespace in a **private** repository. A forwarded port for a private repo
+returns **`302`**, so an inbound-forwarding design cannot work. Instead:
 
-> There is **no** `SATQUERY_UPSTREAM_URL` and **no** `HF_TOKEN` in the live config — the transport is
-> the outbound tunnel, not a forwarded port.
+- the Codespace runs `deploy/codespace/tunnel_agent.py`;
+- the agent **dials out** to `POST /tunnel/agent` and long-polls;
+- work is executed against `http://127.0.0.1:8000` **locally**.
 
-### 3.2 Codespace (inference)
+This inverts the usual direction: the inference host needs **no inbound firewall hole**. It also means
+the transport is only alive while the agent is polling.
+
+**Measured:** `GET /api/health` reported `tunnel.agent_connected: true` with a non-zero `completed`
+counter, and `POST /api/infer {}` returned `422 invalid_request` with the response header
+`x-satquery-transport: tunnel`.
+
+## 5. The full live health payload (VERIFIED, probed)
+
+```json
+{
+  "status": "ok",
+  "service": "satquery-orchestrator",
+  "tunnel": {
+    "agent_connected": true,
+    "agent_id": "codespaces-fd1038",
+    "pending": 0,
+    "completed": 97
+  },
+  "config": {
+    "codespace_name": "potential-space-trout-r4ppw969w45j2pvvw\n",
+    "codespace_port": 8000,
+    "transport_mode": "auto",
+    "tunnel_timeout_s": 150.0,
+    "wake_timeout_s": 120.0,
+    "upstream_timeout_s": 90.0,
+    "device": "cpu",
+    "has_github_token": true
+  }
+}
+```
+
+Note `codespace_name` still carries a **trailing `\n`** — this is **B-02**, cosmetic and `OPEN`; the
+wake path strips it (`_codespace_name()`), so only the health payload reports the raw value.
+
+## 6. Environment variables
+
+### 6.1 Render (gateway) — measured live values
+
+| Variable | Value (live) | Purpose |
+|---|---|---|
+| `CODESPACE_NAME` | `potential-space-trout-r4ppw969w45j2pvvw` | which Codespace to wake |
+| `CODESPACE_PORT` | `8000` | the inference port |
+| `SATQUERY_ALLOWED_ORIGINS` | `https://satquery.pages.dev` | CORS allowlist (never `*`) |
+| `SATQUERY_DEVICE` | `cpu` | device preference |
+| `SATQUERY_TRANSPORT` | `auto` | tunnel first, then forward |
+| `SATQUERY_TUNNEL_TIMEOUT_S` | `150` | how long to wait on the tunnel |
+| `SATQUERY_WAKE_TIMEOUT_S` | `120` | how long to wait for a cold start |
+| `SATQUERY_UPSTREAM_TIMEOUT_S` | `90` | gateway → upstream budget |
+| `GITHUB_TOKEN` | present | Codespace control |
+| `PORT` | platform-supplied | Render's own listen port |
+
+> **Measured absence.** There is **no** `SATQUERY_UPSTREAM_URL` and **no** `HF_TOKEN` in the live
+> config. The transport is the outbound tunnel, not a forwarded port. This contradicts the older
+> `docs/DEPLOYMENT_TOPOLOGY.md` §3.2 table, which predates the tunnel design.
+
+### 6.2 Codespace (inference)
 
 | Variable | Purpose |
 |---|---|
-| `PORT` | platform-assigned; **must be read** |
+| `PORT` | platform-assigned; **must be read** (a historical blocker was that no code read it) |
 | `SATQUERY_DEVICE` | `cpu` \| `cuda` \| `mps` \| `null`; read **without importing torch** |
-| `SATQUERY_MAX_FILE_BYTES` | per-file cap (shared with Render) |
-| `SATQUERY_ASSET_ENABLED` / `SATQUERY_ASSET_DIR` | both required for `/v1/assets`; fails closed (503) otherwise |
+| `SATQUERY_MAX_FILE_BYTES` | per-file cap, shared with Render |
+| `SATQUERY_ASSET_ENABLED` / `SATQUERY_ASSET_DIR` | both required for `/v1/assets`; **fails closed (503)** otherwise |
 | `SATQUERY_ASSET_MAX_FILES` / `SATQUERY_ASSET_TTL_S` | optional handle capacity / lifetime |
 
-## 4. Deploy mechanics
+### 6.3 Config-loader environment overrides
+
+Two registry values can be overridden from the environment without editing the YAML
+(`core/config.py`):
+
+| Variable | Effect |
+|---|---|
+| `SATQUERY_PRECISION` | overrides `training.precision` |
+| `SATQUERY_TORCH_COMPILE` | overrides `deployment.torch_compile` (`"true"` → `True`) |
+
+Both are still validated by the loader — setting `SATQUERY_TORCH_COMPILE=true` **fails startup**
+because finding C-8 forbids `torch.compile`.
+
+## 7. Deploy mechanics
+
+| Tier | Mechanism |
+|---|---|
+| Frontend → Cloudflare Pages | `scripts/stage_pages.mjs` builds a Pages bundle; `npx wrangler pages deploy` |
+| Backend → Render | `render.yaml` blueprint; `main.py` exposes the ASGI object `app` |
+| Inference → Codespace | `deploy/codespace/serve.py` binds `build_space_app()` to `$PORT`; `.devcontainer/` forwards 8000 and starts the tunnel agent via `postStartCommand` |
+| Repository writes | the **GitHub Git Data API** — blob → tree → commit → `PATCH` ref |
+
+### 7.1 Why the Git Data API instead of `git push`
+
+Every deployed file is uploaded as a **blob** whose sha256 is **computed locally and verified against
+the uploaded blob**, then assembled into a tree, committed, and the branch ref patched. This means:
+
+- each file is **content-verified** rather than trusted;
+- deletions are expressed explicitly as `sha: null` tree entries;
+- the deploy is **idempotent** — re-running it with identical content produces no change.
+
+**Measured:** 9 deployed files were re-read from the API and found **sha256 byte-identical** to the
+local copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`).
 
-- **Frontend → Cloudflare Pages.** Staged by `scripts/stage_pages.mjs`, deployed with
-  `npx wrangler pages deploy`.
-- **Backend → Render.** `render.yaml` blueprint; `main.py` exposes `app`.
-- **Inference → Codespace.** `deploy/codespace/serve.py` serves `build_space_app()` on `$PORT`;
-  `.devcontainer/` forwards port 8000 and runs the tunnel agent on start via `postStartCommand`.
-- **Repository writes** are performed through the **GitHub Git Data API** (blob → tree → commit →
-  `PATCH` ref) with **sha256 byte-verification** of every uploaded blob. Deletions are expressed as
-  `sha: null` tree entries. This is used instead of `git push` so each deployed file is verified by
-  content hash.
+## 8. Cold start (documented, not hidden)
 
-### 4.1 Deployed-artifact integrity
+Render's free tier sleeps when idle, and the Codespace may be stopped. Before a request can be served,
+Render must start the Codespace (if stopped) and wait for the tunnel agent to reconnect. The frontend
+shows *"Waking inference engine…"* during this.
 
-Deployed files were re-read from the GitHub API and compared byte-for-byte against the local copies:
-**9 files sha256 byte-identical**, and the deployed HEAD re-read from the API. See
-`verify_deployed_head.py`.
+| Property | Value |
+|---|---|
+| Cold start | **tens of seconds** |
+| Tunnel wait before falling through | `SATQUERY_TUNNEL_TIMEOUT_S` = 150 s |
+| Wake wait | `SATQUERY_WAKE_TIMEOUT_S` = 120 s |
+| Upstream budget | `SATQUERY_UPSTREAM_TIMEOUT_S` = 90 s |
+
+Cold start is **documented rather than papered over**: an honest "this will take a while the first
+time" is better than a silent hang.
+
+### 8.1 The `transport_mode: auto` fallthrough — B-07 (`OPEN`)
+
+`SATQUERY_TRANSPORT=auto` means: **try the tunnel; on timeout, fall through to the forward path.**
+
+The forward path to a **private** repo returns `302` quickly — but the wake step still consumes
+`SATQUERY_WAKE_TIMEOUT_S` (120 s) **first**. So a worst-case failed request takes roughly
+
+```
+150 s (tunnel timeout)  +  120 s (wake timeout on a 302)  ≈  249 s
+```
 
-## 5. Cold start (documented, not hidden)
+This is the **root shape** of the observed transient tunnel gap, and it is why a request can appear to
+hang and then fail. A patch (`fix-b07-forward-unavailable.patch`) was authored and verified
+(`py_compile` clean, applies cleanly to the deployed `main.py`), adding distinct
+`forward_unavailable` (503, terminal 302/401/403 on the forward path) and `upstream_timeout`
+(504, tunnel healthy but slow) codes, plus the `codespace_name` `.strip()` fix.
 
-Render's free tier sleeps when idle and the Codespace may be stopped. The frontend shows
-*"Waking inference engine…"* while Render starts the Codespace and polls health. Cold start is
-therefore **tens of seconds**, and this is documented rather than papered over.
+> **Status: B-07 is `OPEN`.** The patch is **prepared but NOT deployed.** The deployed health payload
+> still shows the trailing `\n`.
 
-## 6. Known deployment issues
+## 9. The five historical backend blockers
 
-| ID | Issue | State |
+Before any backend could boot, five verified blockers had to be closed. The current design closes them:
+
+| # | Blocker (verified) | How it is closed |
 |---|---|---|
-| **B-07** | Transient tunnel-agent gaps → a request can hang or return 504. Patch prepared, **NOT deployed**. | **OPEN** |
-| — | B-07 root shape: in `auto` mode a tunnel timeout **falls through** to the forward path (`main.py:546`), burning `wake_timeout_s = 120` on a `302` (~249 s ≈ 150 + 120). | measured |
-| **B-02** | `/api/health` `codespace_name` trailing `\n`. Cosmetic; the wake path strips it. | **OPEN (cosmetic)** |
-| — | Frontend/Backend/Inference repos are **private**, so their links 404 for an outside audience. | by design |
+| 1 | `requirements.txt` declared no `fastapi` / `uvicorn` / `httpx` / `starlette` | the runtime installs the ASGI stack so `build_space_app()` and the gateway can import |
+| 2 | no code read `$PORT` — a platform port would be ignored | `deploy/codespace/serve.py` binds to `$PORT`; Render reads its own |
+| 3 | hand-rolled CORS raised `405` on `OPTIONS`, so browser preflight failed | the gateway registers `OPTIONS` explicitly / uses Starlette's CORS middleware |
+| 4 | module-level `app = create_app()` swallowed config errors into `app = None` | construction errors now **propagate** (fail-fast) instead of leaving a dead `app` |
+| 5 | adapter integrity unverified on load (`_adapter_sha256` computed but never compared) | the load path compares the digest against an expected value, or fails startup |
+
+## 10. Platform traps (recorded so they are not rediscovered)
+
+| Trap | Detail |
+|---|---|
+| **Cloudflare `_headers` CONCATENATE** | Two matching rules are **merged, not overridden**. A specific rule nested under a broad `/assets/img/*` rule yields `max-age=604800, …, max-age=0, must-revalidate` — and Chromium takes the **FIRST** `max-age`. The file's own "later rules override" comment is **false**. |
+| **Cloudflare 308 redirect** | `X.html` → `/X`. Reference the extensionless path. |
+| **Forwarded port returns 302** | for a private repo — this is *why* the tunnel exists. |
+| **Tunnel agent must start on boot** | via the devcontainer `postStartCommand`, or a restarted Codespace comes up with `agent_connected: false`. |
+| **Never retry `/api/infer` at the gateway** | a retry consumes inference twice. |
+| **`deploy/` is stale and untracked** | not the deployed source. |
 
-## 7. Platform traps (recorded so they are not rediscovered)
+## 11. The superseded design, and what did NOT change
 
-- **Cloudflare `_headers` rules CONCATENATE, they do not override.** Two matching rules are merged;
-  Chromium takes the **first** `max-age` it encounters. A later rule cannot "fix" an earlier one.
-- **Cloudflare 308-redirects `X.html` → `/X`.** Reference the extensionless path.
-- **A forwarded Codespace port returns `302` for a private repo** — which is *why* the tunnel exists.
-- **The tunnel agent must be started by the devcontainer `postStartCommand`**, or a restarted
-  Codespace comes up with `agent_connected:false`.
-- **Never retry `POST /api/infer` at the gateway** — a retry consumes inference twice.
+The earlier design ran inference on an **HF Space with ZeroGPU** (5 GPU-min/day, `@spaces.GPU`
+decoration) behind a **Railway** gateway. The active design changes three things:
+
+1. **CPU-first instead of ZeroGPU.** No code change was required — `device_preference` honours
+   `SATQUERY_DEVICE` and defaults to CPU, every specialist defaults to `device="cpu"`, and all
+   placement is `.to(device)` (never `.cuda()`).
+2. **A real, always-buildable inference environment.** A Codespace gives a reproducible container
+   without a GPU quota or a Space's cold-start constraint. The wake flow replaces ZeroGPU lazy
+   loading as the cold-start story.
+3. **No GPU quota to protect at the gateway.** Rate/size limits remain, but as **fairness** controls
+   rather than quota protection.
+
+**What did NOT change:**
+
+| Unchanged | Detail |
+|---|---|
+| the four-endpoint contract | `/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets` |
+| the gateway responsibility table | §3.2 above |
+| the env-var vocabulary | only host names moved (`SATQUERY_SPACE_URL` → `SATQUERY_UPSTREAM_URL`) |
+| the config freeze | `78f1e3700da15aa1` |
 
-## 8. Preconditions for a fresh deploy (open items)
+`configs/deploy.yaml` still describes the old HF-Space/ZeroGPU target (`platform: huggingface-spaces`,
+`sdk: gradio`, `zerogpu: true`, the `gpu_duration_*` values). It is **frozen paperwork**: no Gradio
+runtime exists in code, and editing it would move `Config.hash`. It is left undisturbed.
+
+## 12. Deployment preconditions (open items)
 
 1. Cloudflare Pages project name / domain (needed for the deploy command and `robots.txt` sitemap).
-2. Artifacts present, or capabilities honestly ship `available: false` (the service **degrades rather
-   than crashes** on absent artifacts).
-3. `HF_TOKEN` set on Render **if** the HF proxy path is used (not used in the live config).
-4. Codespace `.devcontainer/` forwarding `:8000` and starting the tunnel agent.
-5. The five historical backend blockers closed and verified on the first live run (ASGI stack in
-   requirements; `$PORT` read; `OPTIONS` preflight; fail-fast app construction; adapter integrity
-   compared against an expected digest).
-
-## 9. Historical context
-
-The superseded design ran inference on an **HF Space with ZeroGPU** behind a **Railway** gateway. The
-active design moves to **Render + Codespace**, CPU-first, with an **outbound tunnel**. The four-endpoint
-contract, the gateway responsibility table, the env-var vocabulary and the config freeze are
-unchanged — only host names moved. `configs/deploy.yaml` still describes the old HF-Space/ZeroGPU
-target and is left **undisturbed as frozen paperwork** (editing it would move the config hash); no
-Gradio runtime exists in code.
+2. Artifacts present, **or** capabilities honestly ship `available: false` — the service degrades
+   rather than crashes on absent artifacts.
+3. `HF_TOKEN` on Render **if** the HF proxy path is used (not used in the live config).
+4. Codespace `.devcontainer/` forwarding `:8000` **and** starting the tunnel agent.
+5. The five blockers in §9 closed and verified on the first live run.
+
+## 13. What is `NOT RUN` / `OPEN` / `BLOCKED` for this topic
+
+| Item | State |
+|---|---|
+| B-07 tunnel fallthrough | **OPEN** — patch prepared, NOT deployed |
+| B-02 `codespace_name` trailing `\n` | **OPEN** (cosmetic) |
+| A deployed system-level load test | **NOT RUN** |
+| Multi-region / HA deployment | **NOT RUN** |
+| A production database / persistence | **BY DESIGN** (absent) |
+| The ZeroGPU/Gradio deployment target | **REJECTED** (superseded; frozen paperwork only) |
+
+## 14. Where the evidence lives
+
+| Evidence | Location |
+|---|---|
+| Live health + capabilities probe | `release/CURRENT_RELEASE_STATE.md` §1 |
+| Topology contract | `docs/DEPLOYMENT_TOPOLOGY.md`, `docs/DEPLOYMENT_ARCHITECTURE.md` |
+| The undeployed B-07 patch | session scratch: `fix-b07-forward-unavailable.patch` |
+| Deployed-artifact integrity check | `verify_deployed_head.py` (session scratch) |
+| Live validation (3 passes) | `.workbuddy-ai/scratch/live_validation/` |