release: add docs/OPERATIONS.md

docs/OPERATIONS.md

@@ -0,0 +1,1185 @@
+# Operations Manual
+
+**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `ATTEMPTED` · `NOT RUN` · `BLOCKED` ·
+`DEFERRED` · `OPEN` · `RESOLVED` · `BY DESIGN`.
+
+This is the operator-facing manual for the **live SatQuery AI stack**. It answers four questions that
+the architecture chapters deliberately do not: *what is running, right now, and who owns it*; *how do I
+bring it up and keep it up*; *how do I tell a transient transport gap from a real failure*; and *what
+monitoring, capacity and cost machinery is **absent** so I do not assume it exists*.
+
+It is written for the person who has to make the system answer a question in front of an audience, and
+for the person who has to diagnose it at 23:00 when it does not.
+
+> **Read this first.** The live system runs across **three private repositories** plus the public
+> umbrella. The monorepo working copy — including the `deploy/` directory inside it — is **not** the
+> deployed source. `deploy/` in the monorepo is **stale and untracked** (`git status` reports
+> `?? deploy/`; verified in the working copy). Any operational fix must be applied to the real
+> repositories, never to the monorepo copy (`docs/DEPLOYMENT.md` §1, §10).
+
+> **Second rule.** Two known defects are `OPEN` and are **not** fixed in production: **B-07** (transient
+> tunnel gaps) and **B-02** (a cosmetic trailing newline in one health field). Neither may be described
+> as resolved, and the B-07 patch is **prepared but NOT deployed**. This document never upgrades them.
+
+---
+
+## Table of contents
+
+**Part I — The operational model**
+1. What runs where
+2. The tier inventory, with the deployed revision of each
+3. Who owns what
+4. The operational invariants (four rules that must never be broken)
+5. The stale-copy problem, in operational terms
+
+**Part II — The runbook**
+6. Warm the stack before a demo
+7. Restart after an idle-stop
+8. Tell a tunnel gap from a real failure
+9. What the client shows while waking
+
+**Part III — Cold start and the timing budget**
+10. The cold-start shape
+11. The four timeouts, and why the relationship matters
+12. The worst case: ≈249 s under B-07
+
+**Part IV — The known defects, in operational terms**
+13. B-07 — transient tunnel gaps (`OPEN`)
+14. B-02 — the trailing newline (`OPEN`, cosmetic)
+
+**Part V — Monitoring and alerting**
+15. What exists
+16. What does **not** exist
+17. Why "no alerting" is a design fact, not an oversight
+
+**Part VI — Capacity and cost**
+18. The capacity shape
+19. The cost shape, and the one absence that has a code artifact
+
+**Part VII — Incident triage**
+20. The symptom → cause → check → action table
+21. The consolidated decision tree
+22. The machine-code reference
+
+**Part VIII — Routine maintenance**
+23. Rotating credentials (procedure only)
+24. Restarting the tunnel agent
+25. Re-warming the model cache
+26. Deploying a change (the Git Data API path)
+
+**Part IX — Known operational gaps**
+27. The explicit gaps list
+28. `NOT RUN` / `OPEN` / `BLOCKED` / `UNKNOWN` for operations
+
+**Part X — Evidence**
+29. Where the evidence lives
+
+---
+
+# Part I — The operational model
+
+## 1. What runs where
+
+The live stack is four tiers in a straight line, plus a model tier that is reached *through* the
+inference tier rather than by the user (`release/repo/docs/DEPLOYMENT.md` §2):
+
+```
+Browser
+  │  HTTPS
+  ▼
+Cloudflare Pages  —  satquery.pages.dev                 (static frontend, 11 pages)
+  │  HTTPS / JSON  →  /api/*
+  ▼
+Render            —  satquery-backend-m4yv.onrender.com  (orchestrator / API gateway)
+  │  outbound long-poll  POST /tunnel/agent
+  ▼
+GitHub Codespace  —  FastAPI inference, CPU, port 8000
+  │  build_space_app()
+  ▼
+specialists: SmolVLM · RemoteCLIP · MiniLM · CROMA · STANet
+  │
+  ▼
+ResultEnvelope  →  tunnel  →  Render  →  browser
+```
+
+```mermaid
+flowchart LR
+  U[Browser] -->|HTTPS| CF["Cloudflare Pages<br/>static frontend"]
+  CF -->|"HTTPS JSON<br/>/api/health · /api/capabilities · /api/infer · /api/assets"| R["Render<br/>orchestrator / gateway"]
+  R -->|"outbound long-poll<br/>POST /tunnel/agent"| C["GitHub Codespace<br/>FastAPI inference :8000"]
+  C --> S[(SmolVLM · RemoteCLIP<br/>MiniLM · CROMA · STANet)]
+  C -->|ResultEnvelope| R
+  R -->|"envelope + error translation"| CF
+```
+
+Three properties of this diagram matter operationally, and each is the subject of a section below:
+
+1. **The transport is an outbound tunnel, not an inbound port.** The Codespace dials *out* to Render.
+   Render never dials into the Codespace. The transport is therefore alive only while an agent process
+   is polling — which is why "is the agent connected?" is the single most important operational
+   question (§6, §8).
+2. **There is exactly one inference host.** One Codespace, one Render service, no replicas, no
+   autoscaling (`render.yaml` declares a single web service with `plan: free`; plan §74 lists
+   `autoscaling` under **Not included**). Capacity is therefore bounded by that one host (§18).
+3. **Inference is CPU-only.** `SATQUERY_DEVICE=cpu` is set on both the orchestrator and the Codespace
+   (`render.yaml`, `.devcontainer/devcontainer.json`), and every specialist defaults to `device="cpu"`
+   (`docs/DEPLOYMENT_DECISION.md` §5). No GPU path is on the live critical path.
+
+## 2. The tier inventory, with the deployed revision of each
+
+Read from the GitHub API during the release reconnaissance (`release/CURRENT_RELEASE_STATE.md` §1;
+`release/repo/docs/DEPLOYMENT.md` §1):
+
+| Component | Repository | Visibility | Branch | Revision | Host |
+|---|---|---|---|---|---|
+| Frontend | `Anish-lab-blip/SatQuery-Frontend` | **private** | `main` | **`2d7ae53b482d`** | Cloudflare Pages → `satquery.pages.dev` |
+| Backend / orchestrator | `Anish-lab-blip/SatQuery-Backend` | **private** | `main` | **`89d80eaddec5`** | Render → `satquery-backend-m4yv.onrender.com` |
+| Inference | `Anish-lab-blip/SatQuery-Inference` | **private** | `main` | **`5a0936ace491`** | Codespace `potential-space-trout-r4ppw969w45j2pvvw`, port 8000, via outbound tunnel |
+| Public umbrella | `Anish-lab-blip/SatQuery-AI` | **public** | `main` | `3dcabd32da41` ("Initial commit") | this release home |
+| Monorepo (working copy) | `C:/Users/anish/satquery-ai` | local only | `master` | `9d57aed` | **no git remote**; 334 dirty entries |
+| Hugging Face | `thundercode/SatQuery` | **public** | `main` | lastModified `2026-09-25T16:26:53Z` | model tier |
+
+The three private repositories are private **by design**; their links return 404 for an outside
+audience (`release/CURRENT_RELEASE_STATE.md` §6). An operator therefore cannot browse the deployed
+source from a public URL — the deployed files must be fetched with an authenticated API call
+(`release/repo/docs/DEPLOYMENT.md` §7.1, §10).
+
+> **The monorepo's `deploy/` is not the deployed source.** This is the single most important trap in
+> the whole system (§5).
+
+## 3. Who owns what
+
+The ownership table below is derived from the code and the deployment records. "Owner" means *the
+person or role that must act when this tier misbehaves*.
+
+| Tier | Owner | What they own | What they must never do |
+|---|---|---|---|
+| Cloudflare Pages (frontend) | Frontend maintainer | the static bundle, `_headers`, `robots.txt`, the Analyze console | add a server-side secret — the tier holds none |
+| Render (gateway) | Backend maintainer | the orchestrator revision, the env-var set, the CORS allowlist, the tunnel hub state | retry `POST /api/infer` (§4) |
+| Codespace (inference) | Inference maintainer | the Codespace, the tunnel agent, the asset directory, the HF cache | let a stale serve process keep answering (§5) |
+| Hugging Face (model tier) | Release owner | model cards, the pinned model references, the released checksums | treat the Hub as the runtime inference host — it is not |
+| Credentials | Owner (human) | the GitHub PAT and the account tokens | record any credential value in a public document (§23) |
+
+Two decisions are explicitly **not** an agent's to make, and both gate operational change
+(`docs/PHASE19_FINAL_HARDENING.md` §7): the **SDK choice** (irrelevant on the live path, but still
+unmade for the frozen manifest) and the **rate-limit / size-limit values** (which bound one client's
+share of capacity). Neither is needed to operate the system as deployed.
+
+## 4. The operational invariants
+
+Four rules are load-bearing. Each is enforced somewhere in code or config, and each has a documented
+failure if broken.
+
+### 4.1 The config hash is frozen
+
+`Config.hash == 78f1e3700da15aa1`. The loader computes a sha256 over the whole registry
+(`core/config.py:76-80`) and every evaluation run records it. **Editing `configs/base.yaml` moves the
+hash and invalidates every artifact keyed to it** (`release/repo/docs/DEPLOYMENT.md` §11;
+`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §6.1). Deployment state that must *not* move the hash — asset-store
+capacity, TTL, the per-file cap — is read from the **environment**, not from the YAML
+(`release/repo/README.md` §Installation).
+
+Operational consequence: **never edit `configs/base.yaml` to point at a deployment artifact.** The
+serving path wires checkpoints through the registry's `builders=` override precisely so it does not have
+to (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §6.1).
+
+### 4.2 The gateway never retries `POST /api/infer`
+
+> *"Render must not retry `POST /api/infer` on its own — a retry would consume inference a second
+> time. The client decides on retry."* (`docs/DEPLOYMENT_TOPOLOGY.md` §2)
+
+This is stated in three places (`docs/DEPLOYMENT_TOPOLOGY.md` §2, `docs/DEPLOYMENT.md` §7,
+`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §4.3) because a retry is the natural thing to add and the wrong
+thing to add. On the live CPU deployment it wastes compute; on the historical ZeroGPU target it spent a
+metered GPU-minute twice.
+
+### 4.3 The CORS allowlist is explicit and never a wildcard
+
+The gateway assembles its allowlist from `SATQUERY_ALLOWED_ORIGINS` plus a hard-coded production origin
+plus a fixed list of development origins (`deploy/render/main.py:139-216`). A `*` raises
+(`deploy/render/main.py:204-208`). The live value is `https://satquery.pages.dev`
+(`release/repo/docs/DEPLOYMENT.md` §6.1).
+
+Operational consequence: a new frontend origin must be **added** to the env var; it will not work by
+accident.
+
+### 4.4 One inference host, one asset store
+
+There is one Codespace, and its filesystem is **ephemeral** (`deploy/codespace/launch.sh:44-47`). Uploaded
+assets are written under `SATQUERY_ASSET_DIR` (default `/tmp/satquery-assets`) and are TTL'd (900 s
+default). A Codespace restart empties the store and makes every previously issued handle unresolvable
+(`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §3.1.1).
+
+Operational consequence: a handle that worked seconds ago may return `400 input_error` after a restart.
+That is documented behaviour, not a bug (§20).
+
+## 5. The stale-copy problem, in operational terms
+
+Three copies of the deployment code exist, and confusing them is the most expensive operational
+mistake in the system.
+
+| Copy | What it is | Trustworthy? |
+|---|---|---|
+| the monorepo `deploy/` | local, **untracked** (`?? deploy/`), stale | **no** — it is not the deployed source |
+| the session scratch copy | a local copy used to author and verify the B-07 patch | **no** — it is "deployed + patch", not deployed |
+| the private repositories | the real deployed source | **yes** — fetch it before editing |
+
+Evidence for the divergence is direct. The monorepo's `deploy/render/main.py` (532 lines) exposes
+`/api/health` with a `config` block that has **no** `tunnel` field and **no** `transport_mode`,
+`tunnel_timeout_s` or `wake_timeout_s` keys (`deploy/render/main.py:444-466`), whereas the **live**
+payload carries all of them (`release/repo/docs/DEPLOYMENT.md` §5). The monorepo copy also contains no
+`tunnel_agent.py` and no `doctor.sh`, even though `deploy/codespace/launch.sh` invokes both
+(`deploy/codespace/launch.sh:83,89,153,160-168,179`). The two are different programs.
+
+> **Operational rule.** Before changing anything, fetch the deployed `main.py` from the private
+> repository and diff it against what you are about to edit. The monorepo copy will silently disagree.
+
+The `deploy/codespace/launch.sh` file *is* useful as documentation of intent — its header explains why
+it is defensive (`deploy/codespace/launch.sh:14-24`) — but it is a copy, and its references to
+`tunnel_agent.py` resolve only in the deployed repository.
+
+---
+
+# Part II — The runbook
+
+Every step in this part is grounded in a file. Commands are quoted as they appear in the sources.
+
+## 6. Warm the stack before a demo
+
+Three things can be cold, and all three are warmed differently (`docs/architecture/10-observability-and-ops.md`
+§7.1):
+
+| What is cold | How it warms | Bound |
+|---|---|---|
+| the Render orchestrator | the first request to any `/api/*` route | Render free tier sleep/wake cycle |
+| the Codespace | `ensure_codespace_up()` starts it and polls | `wake_timeout_s = 120` |
+| the HF model cache | `warm_cache.py`, or the first model-touching request | one download per model |
+
+### Step 1 — confirm the tunnel agent is connected
+
+This is the check that matters most, because the tunnel is the live transport and the forwarded-port
+path is dead for a private repository (`release/CURRENT_RELEASE_STATE.md` §2: *"The forwarded-port path
+is **dead** (302 for a private repo); the tunnel is the live transport."*).
+
+```bash
+curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/health
+```
+
+Read `tunnel.agent_connected`. `true` means an agent has polled recently; `false` means **no agent has
+polled recently**, and under `transport_mode: auto` a request will then take the forward path, which for
+a private repository fails after burning the wake timeout (`docs/architecture/10-observability-and-ops.md`
+§7.1).
+
+> **`--noproxy '*'` is not optional in the authoring sandbox.** The sandbox proxy is dead; without the
+> flag the request fails before reaching Render. In a normal environment the flag is harmless
+> (`release/repo/docs/REPRODUCIBILITY.md` §10.1).
+
+### Step 2 — if `agent_connected` is false, start the Codespace
+
+The agent is launched by the devcontainer's `postStartCommand`, which runs `launch.sh`:
+
+```json
+"postStartCommand": "bash deploy/codespace/launch.sh"
+```
+
+(`.devcontainer/devcontainer.json:18`)
+
+Starting the Codespace is what re-runs `postStartCommand` and therefore reconnects the agent (§7).
+`launch.sh` is defensive by design, and its own header explains why:
+
+> *"`setsid` alone is NOT enough in Codespaces. The lifecycle shell that runs postStartCommand can still
+> reap the process group, which showed up in production as 'the agent announced once, then vanished' —
+> the hub then reported agent_connected=false and /api/infer fell back to the dead forwarded-port path
+> (401 -> wake_timeout)."* (`deploy/codespace/launch.sh:14-19`)
+
+The script therefore uses `setsid + nohup + </dev/null` plus a **supervising wrapper** that restarts the
+agent if it exits (`deploy/codespace/launch.sh:20-23,160-168`), and then **verifies** the agent came up:
+
+```bash
+sleep 4
+
+if ! pgrep -f "deploy/codespace/tunnel_agent.py" > /dev/null 2>&1; then
+  echo "WARNING: the tunnel agent is not running. Last log lines:" >&2
+  ...
+else
+  echo "tunnel agent process is up (pid $(pgrep -f 'deploy/codespace/tunnel_agent.py' | head -1))"
+  if grep -q "announced to hub" "$TUNNEL_LOG" 2>/dev/null; then
+    echo "tunnel agent announced to the hub successfully"
+  ...
+```
+
+(`deploy/codespace/launch.sh:177-191`)
+
+The verification exists because of a specific failure:
+
+> *"Backgrounding with all output discarded means a crashing agent is completely invisible — that is
+> exactly how a missing `httpx` hid itself."* (`deploy/codespace/launch.sh:174-176`)
+
+### Step 3 — warm the HF cache, if the Codespace was rebuilt
+
+`warm_cache.py` pre-downloads the four pinned models:
+
+```bash
+python deploy/codespace/warm_cache.py
+```
+
+It reports per-model `OK` / `SKIPPED` / `FAILED`, never aborts on a single miss, and **always exits 0** so
+a cache miss cannot fail a build (`deploy/codespace/warm_cache.py:8-10,110-112`). The four models and
+their pinned revisions are transcribed verbatim from `configs/base.yaml`
+(`deploy/codespace/warm_cache.py:12-16,35-60`):
+
+| key | repo | revision | file |
+|---|---|---|---|
+| `vlm` | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | (snapshot) |
+| `grounding` | `chendelong/RemoteCLIP` | `bf1d8a3ccf2d` | `RemoteCLIP-ViT-B-32.pt` |
+| `router` | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | (snapshot) |
+| `croma` | `antofuller/CROMA` | `0dd28e3d633b` | `CROMA_base.pt` |
+
+It is idempotent and safe to re-run; a second run is a no-op because the blob is already on disk
+(`deploy/codespace/warm_cache.py:3-6`). Two environment switches skip work:
+`SATQUERY_WARM_OFFLINE` / `HF_HUB_OFFLINE` (skip everything) and `SATQUERY_WARM_SKIP="croma,grounding"`
+(skip named models) (`deploy/codespace/warm_cache.py:23-25`).
+
+> **Note.** `warm_cache.py` warms four models. The **fifth** pinned backbone in the specialist stack —
+`STANet`/change — is a local trained head, not a Hub backbone, and is not in the warm list. If the change
+head is absent the capability degrades honestly rather than failing the warm step.
+
+### Step 4 — run one throwaway analysis
+
+A single cheap query confirms the whole chain end to end. Do this **before** the demo, not during it.
+Watch for: a `run_id`, a `mock_nodes` count of 0, and an answer carrying a `[task]` tag
+(`release/repo/docs/REPRODUCIBILITY.md` §6.3).
+
+### The warm-state checklist
+
+| Check | Command | Expected |
+|---|---|---|
+| orchestrator up | `curl --noproxy '*' .../api/health` | `status: ok`, `service: satquery-orchestrator` |
+| tunnel connected | same payload | `tunnel.agent_connected: true` |
+| capabilities | `curl --noproxy '*' .../api/capabilities` | six tasks, all `available: true` |
+| one live run | drive the Analyze console | a `run_id`, `mock_nodes: 0` |
+
+## 7. Restart after an idle-stop
+
+A Codespace stops after an idle period. When it stops, the tunnel agent stops polling, and
+`GET /api/health` reports `tunnel.agent_connected: false`
+(`docs/DEPLOYMENT_TOPOLOGY.md` §2).
+
+**The reconnect is automatic on start, because `postStartCommand` runs `launch.sh`.** The chain is:
+
+```
+Codespace start
+  → devcontainer postStartCommand:  bash deploy/codespace/launch.sh      (.devcontainer/devcontainer.json:18)
+  → launch.sh: start serve.py on $PORT (if not already current)          (launch.sh:110-147)
+  → launch.sh: start the supervised tunnel agent -> $SATQUERY_HUB_URL     (launch.sh:149-169)
+  → launch.sh: sleep 4, verify the agent process and the announce line    (launch.sh:171-191)
+  → the agent dials POST /tunnel/agent and long-polls                     (launch.sh:150-156)
+  → GET /api/health: tunnel.agent_connected becomes true
+```
+
+The hub URL the agent dials is `SATQUERY_HUB_URL`, defaulting to
+`https://satquery-backend-m4yv.onrender.com` (`deploy/codespace/launch.sh:59-61`).
+
+**What the operator does:**
+
+1. Start the Codespace (or let the wake path start it — `ensure_codespace_up()` calls the GitHub
+   Codespaces `POST .../start` API when `state != "available"`, `deploy/render/main.py:299-356`).
+2. Wait for `postStartCommand` to run.
+3. Re-read `GET /api/health` and confirm `tunnel.agent_connected: true` (§6 step 1).
+
+**Two things that make a restart go wrong, and their mitigation:**
+
+| Failure | Symptom | Mitigation in `launch.sh` |
+|---|---|---|
+| The agent is launched but immediately reaped by the lifecycle shell | agent announces once, then vanishes; hub reports `agent_connected: false`; `/api/infer` falls back to the dead forward path (401 → wake_timeout) | `setsid + nohup + </dev/null` plus a supervising restart loop (`launch.sh:14-24,160-168`) |
+| A **stale** serve process keeps answering from OLD code | `/v1/health` and `/v1/capabilities` answer, but from the previous revision's capabilities | a **stamp** recording the revision + asset config; a mismatch restarts the server (`launch.sh:100-147`) |
+
+> *"A stale serve process is worse than no process: it answers /v1/health and /v1/capabilities from OLD
+> code, so the deployment looks alive while reporting the previous revision's capabilities."*
+> (`deploy/codespace/launch.sh:111-113`)
+
+The stamp is the closest thing in the system to a deployment-identity check:
+
+```bash
+_current_stamp() {
+  printf 'rev=%s asset_enabled=%s asset_dir=%s\n' \
+    "$(git rev-parse HEAD 2>/dev/null || echo nogit)" \
+    "${SATQUERY_ASSET_ENABLED:-}" \
+    "${SATQUERY_ASSET_DIR:-}"
+}
+```
+
+(`deploy/codespace/launch.sh:103-108`)
+
+It is **local to the Codespace** and is not exposed on any HTTP route — an operator on the orchestrator
+side cannot see it (`docs/architecture/10-observability-and-ops.md` §7.5).
+
+### The preflight that refuses a half-configured start
+
+`launch.sh` refuses to start if the Python dependencies or the `app` package cannot be imported
+(`deploy/codespace/launch.sh:70-91`). The dependency check is explicit about `httpx`, because a missing
+`httpx` once made the agent die instantly and the supervised loop hid the error in a log file
+(`deploy/codespace/launch.sh:76-79`):
+
+```bash
+if ! python -c "import yaml, pydantic, fastapi, uvicorn, httpx" 2>/dev/null; then
+  echo "ERROR: Python deps are missing (need yaml, pydantic, fastapi, uvicorn, httpx)." >&2
+  ...
+  exit 1
+fi
+```
+
+> **Caveat — the referenced repair tool is not in this tree.** `launch.sh` points the operator at
+> `bash deploy/codespace/doctor.sh --install` (`launch.sh:83,89,182`), but **`doctor.sh` does not exist in
+> the monorepo working copy**, and neither does `tunnel_agent.py` (§5). Whether `doctor.sh` exists in the
+> deployed `SatQuery-Inference` repository is `UNKNOWN — not established from the available evidence`.
+
+## 8. Tell a tunnel gap from a real failure
+
+This is the runbook's most useful procedure, and it is grounded in a measured timing.
+
+### The signature
+
+A request that hangs for **≈249 seconds** and then returns **`504`** is the tunnel-gap signature, not a
+broken model. The arithmetic is exact:
+
+```
+tunnel_timeout_s (150) + wake_timeout_s (120) = 270 s  (nominal)
+measured                                            ≈ 249 s
+```
+
+> *"B-07 root shape: in `auto` mode a tunnel timeout **falls through** to the forward path
+> (`main.py:546`), burning `wake_timeout_s = 120` on a `302` (~249 s ≈ 150 + 120)."*
+> (`release/CURRENT_RELEASE_STATE.md` §6; `release/repo/docs/DEPLOYMENT.md` §8.1)
+
+### The three signals to read, in order
+
+1. **`tunnel.agent_connected` on a fresh `/api/health`.** If `false`, it is a tunnel gap and the remedy is
+   §6 step 2. Do not trust a single reading — the flag is a freshness window, so re-read.
+2. **The elapsed time.** ≈249 s is the B-07 signature. A fast failure is something else.
+3. **The machine code.** The codes are disjoint and each implies a different action (§22).
+
+### The decision tree
+
+```mermaid
+flowchart TD
+    A["a request hung, or returned 5xx"] --> B{"re-read /api/health<br/>tunnel.agent_connected?"}
+    B -->|"true"| C{"was the wait ≈249 s?"}
+    B -->|"false"| D["TUNNEL GAP<br/>the agent is not polling.<br/>Start the Codespace (§6)."]
+    C -->|"yes, 504"| E["TUNNEL GAP<br/>agent went stale mid-request,<br/>or the Codespace stopped.<br/>B-07. Mitigate operationally."]
+    C -->|"no"| F{"what was the code?"}
+    F -->|"invalid_request 422"| G["a CLIENT bug — the body<br/>did not match AnalysisRequest"]
+    F -->|"model_load_error / model_unavailable"| H["an ARTIFACT defect —<br/>surface it, do not retry blindly"]
+    F -->|"upstream_timeout 504"| I["tunnel healthy but slow —<br/>the agent did not complete in 150 s"]
+    F -->|"other"| J["read the code and<br/>trace.errors[]"]
+```
+
+> **Note which branches exist only in the patch.** `forward_unavailable` and `upstream_timeout` do **not**
+> exist in the deployed revision (§13). An operator on the deployed system will not see them; they will
+> see `wake_timeout` after ≈249 s instead.
+
+## 9. What the client shows while waking
+
+The frontend is not silent during a cold start. It shows *"Waking inference engine…"* while Render starts
+the Codespace (`docs/DEPLOYMENT_TOPOLOGY.md` §2, §2 mermaid; `release/repo/docs/DEPLOYMENT.md` §8).
+
+The orchestrator's side of this is a response header. The wake path is **blocking wake-then-proxy**: the
+client waits and receives the result, and the response is tagged:
+
+```python
+out.headers["X-SatQuery-State"] = "waking" if woke else "ready"
+```
+
+(`deploy/render/main.py:486-489`)
+
+The module docstring states the intent plainly:
+
+> *"The client simply **waits** (blocking, wake-then-proxy) and receives the result; the frontend
+> independently shows 'Waking inference engine...' on slow responses. We optionally tag the response with
+> `X-SatQuery-State: waking` so the frontend can confirm the delay was a cold start."*
+> (`deploy/render/main.py:13-20`)
+
+So an operator watching a demo sees: the client's "Waking inference engine…" message, a wait, and then
+either a result or a `504`. **A `504` after ≈249 s is the B-07 shape, not a broken model** (§8).
+
+---
+
+# Part III — Cold start and the timing budget
+
+## 10. The cold-start shape
+
+The cold start is **blocking and documented**, not hidden. The shape an operator should expect
+(`docs/architecture/10-observability-and-ops.md` §7.2):
+
+| Phase | What happens | Bound |
+|---|---|---|
+| 0 | the client sends `POST /api/infer` and **waits** | — |
+| 1 | `GET` the Codespace via the GitHub API; `POST .../start` if not `available` | one API round trip |
+| 2 | poll `GET {base}/v1/health` every 2 s, each with a 10 s timeout | ≤ `wake_timeout_s = 120` |
+| 3 | on the first `200`, proxy the original request | ≤ `upstream_timeout_s = 90` |
+| 4 | the response carries `X-SatQuery-State: waking` | — |
+
+The poll interval and per-poll timeout are module constants:
+
+```python
+# Polling knobs for the wake loop.
+_WAKE_POLL_INTERVAL_S = 2.0
+_WAKE_HEALTH_TIMEOUT_S = 10.0
+```
+
+(`deploy/render/main.py:76-78`)
+
+> **Do not quote a single cold-start number.** The documented statement is *"tens of seconds"*
+> (`release/repo/docs/DEPLOYMENT.md` §8), and the bounded worst case is `wake_timeout_s = 120` before a
+> `504`. **No measured cold-start distribution exists:**
+> `UNKNOWN — not established from the available evidence`
+> (`docs/architecture/10-observability-and-ops.md` §7.2).
+
+## 11. The four timeouts, and why the relationship matters
+
+The live payload reports three of them; the fourth comes from the frozen config.
+
+| Timeout | Value | Where it lives | What it bounds |
+|---|---|---|---|
+| `tunnel_timeout_s` | **150.0 s** | live config (`release/repo/docs/DEPLOYMENT.md` §5) | how long a request waits on the tunnel before falling through |
+| `wake_timeout_s` | **120.0 s** | live config | how long the wake loop waits for `/v1/health` |
+| `upstream_timeout_s` | **90.0 s** | live config | the gateway → upstream proxy budget |
+| `agent.timeout_seconds` | **120 s** | `configs/base.yaml` (`agent.timeout_seconds: 120`) | the Space's own total request budget |
+
+The **relationship** between the last two is the one that must not be inverted
+(`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §4.2):
+
+```
+gateway upstream timeout  <  agent.timeout_seconds  ≤  the Space's own request budget
+         90 s             <         120 s
+```
+
+Both failure directions are documented:
+
+- **Gateway timeout too short** — it kills a legitimately running `grounding` or `optical_sar` call and
+  reports it as an upstream failure. The Space's error, not the client's.
+- **Gateway timeout too long** — it holds a connection past the point the Space itself has given up,
+  converting a clean upstream timeout into a client-side hang.
+
+> The lower bound of the *historical* window was 45 s (the longest single `gpu_duration_*`). On the live
+> **CPU** deployment the ZeroGPU durations are frozen paperwork (§19), so the binding upper constraint is
+> the 120 s agent timeout and the live gateway value is 90 s.
+
+## 12. The worst case: ≈249 s under B-07
+
+`SATQUERY_TRANSPORT=auto` means **try the tunnel; on timeout, fall through to the forward path**
+(`release/repo/docs/DEPLOYMENT.md` §8.1). The forward path to a private repository returns `302` quickly,
+but the wake step still consumes `SATQUERY_WAKE_TIMEOUT_S` (120 s) first. So a worst-case failed request
+takes roughly:
+
+```
+150 s (tunnel timeout)  +  120 s (wake timeout on a 302)  ≈  249 s
+```
+
+This is the **root shape** of the observed transient tunnel gap, and it is why a request can appear to
+hang and then fail (`release/repo/docs/DEPLOYMENT.md` §8.1). It is `OPEN` (§13).
+
+---
+
+# Part IV — The known defects, in operational terms
+
+## 13. B-07 — transient tunnel gaps (`OPEN`)
+
+**B-07 is `OPEN`.** The patch is prepared and **not deployed**. This is the single most important
+operational fact in this manual.
+
+> *"B-07 | Transient tunnel-agent gaps → a request can hang or return 504. Patch prepared, **NOT
+> deployed**. | **OPEN**"* (`release/CURRENT_RELEASE_STATE.md` §6)
+
+### What an operator experiences
+
+- The tunnel agent is briefly absent (a restart, a reap, a gap).
+- A request issued during the gap either hangs or returns `504`.
+- In `auto` mode the hang lasts up to ≈249 s before the `504` (§12).
+- Once the agent reconnects, the next request succeeds.
+
+### The root cause, exactly
+
+In `auto` mode a tunnel timeout **falls through to the forward path** (`main.py:546`), and the forward
+path to a private repository returns `302`. The wake step burns `wake_timeout_s = 120` on that `302`
+before the request fails (`release/CURRENT_RELEASE_STATE.md` §6).
+
+### What the patch does
+
+The patch was authored and verified (`py_compile` clean, applies cleanly to the deployed `main.py`)
+(`release/repo/docs/DEPLOYMENT.md` §8.1). It makes three changes:
+
+| Change | Code | Effect |
+|---|---|---|
+| A | `forward_unavailable` (`503`, `recoverable: true`) on a **terminal** `302`/`401`/`403` | converts a 504-after-249 s into a 503-early with an actionable code |
+| B | `upstream_timeout` (`504`) for "tunnel healthy but slow" | distinguishes a slow agent from a dead forward path |
+| C | `/api/health` `codespace_name` `.strip()` | fixes B-02 |
+
+### The honesty note attached to the patch
+
+> *"The report records that an earlier claim that the patch 'would not have prevented' the observed 504
+> 'was wrong and was retracted'. The corrected position: 'Change A is genuinely **on the failing path** —
+> it converts a 504-after-249 s into a 503-early with an actionable code.'"*
+> (`release/CURRENT_RELEASE_STATE.md` §8 evidence list; `docs/architecture/02-deployment-topology.md` §6.4)
+
+Do not repeat the retracted version.
+
+### Why it is not deployed
+
+> *"the patch is not needed for the demo and touches the live backend. The residual is better mitigated
+> operationally (keep the Codespace warm, raise the idle timeout)."*
+> (`DELIVERY_REPORT_2026-09-25.md` §4, quoted in `docs/architecture/10-observability-and-ops.md` §7.4)
+
+### The operational mitigation (this is what an operator actually does)
+
+1. Keep the Codespace **warm** before and during any demo (§6).
+2. **Raise the Codespace idle timeout** so it does not stop mid-session.
+3. Re-read `/api/health` before a run, and treat `agent_connected: false` as "warm the stack now".
+4. Expect a ≈249 s hang followed by a `504` if the agent goes stale mid-request — and **retry**, because
+   the residual is transient.
+
+> **Do not upgrade B-07.** It is `OPEN`. It is not `RESOLVED`, and the patch is not deployed — the live
+> payload's `codespace_name` trailing `\n` is the witness (§14).
+
+## 14. B-02 — the trailing newline (`OPEN`, cosmetic)
+
+`GET /api/health` reports the Codespace name with a trailing newline:
+
+```json
+"codespace_name": "potential-space-trout-r4ppw969w45j2pvvw\n"
+```
+
+(`release/repo/docs/DEPLOYMENT.md` §5; `release/CURRENT_RELEASE_STATE.md` §1)
+
+**It is cosmetic.** The wake path strips it — `_codespace_name()` calls `.strip()` before using the value
+(`deploy/render/main.py:99-106`) — so only the health payload reports the raw value
+(`release/repo/docs/DEPLOYMENT.md` §5).
+
+**It is `OPEN`.** Its presence is also the operational **witness** that the B-07 patch is not deployed:
+change C of that patch is the `.strip()` fix, so a live payload still showing the trailing `\n` proves the
+patch is absent (`docs/architecture/10-observability-and-ops.md` §7.4).
+
+> **Do not "fix" B-02 by editing the health payload on the live service.** The fix ships with the B-07
+> patch, which is deliberately not deployed. A cosmetic newline is not worth a live-backend change on
+> its own.
+
+---
+
+# Part V — Monitoring and alerting
+
+## 15. What exists
+
+SatQuery AI observes **exactly three things** (`docs/architecture/10-observability-and-ops.md` §6): whether
+the transport is up, what one run did, and what failed inside the server.
+
+### 15.1 The health payload
+
+The measured live payload (`release/repo/docs/DEPLOYMENT.md` §5):
+
+```json
+{
+  "status": "ok",
+  "service": "satquery-orchestrator",
+  "tunnel": {
+    "agent_connected": true,
+    "agent_id": "codespaces-fd1038",
+    "pending": 0,
+    "completed": 97
+  },
+  "config": {
+    "codespace_name": "potential-space-trout-r4ppw969w45j2pvvw\n",
+    "codespace_port": 8000,
+    "transport_mode": "auto",
+    "tunnel_timeout_s": 150.0,
+    "wake_timeout_s": 120.0,
+    "upstream_timeout_s": 90.0,
+    "device": "cpu",
+    "has_github_token": true
+  }
+}
+```
+
+Field by field, for the operator:
+
+| Field | Meaning | Operational use |
+|---|---|---|
+| `status` | orchestrator liveness | a single up/down bit for the gateway tier |
+| `tunnel.agent_connected` | an agent polled within the freshness window | **the** check before a demo (§6) |
+| `tunnel.agent_id` | which agent | identifies the Codespace agent that is connected |
+| `tunnel.pending` | in-flight tunnel requests | a rising value means work is queueing |
+| `tunnel.completed` | a monotonic delivery counter, per Render process | trend only — it **resets on restart** |
+| `config.codespace_name` | the target Codespace | carries B-02's trailing `\n` (§14) |
+| `config.codespace_port` | the inference port | `8000` |
+| `config.transport_mode` | `auto` / `tunnel` / `forward` | `auto` is what makes B-07 reachable (§13) |
+| `config.tunnel_timeout_s` | the tunnel wait | `150.0` (§11) |
+| `config.wake_timeout_s` | the cold-start wait | `120.0` (§11) |
+| `config.upstream_timeout_s` | the proxy budget | `90.0` (§11) |
+| `config.device` | device preference | `cpu` |
+| `config.has_github_token` | whether a token is present (boolean only) | a `false` here means the wake path cannot start the Codespace |
+
+> **`completed` was measured at three different values** across probes (`97`, `314`, and others). It is a
+> counter that resets when the Render process restarts, not a constant. **Do not treat any single reading
+> as the value** (`docs/architecture/10-observability-and-ops.md` §2.3;
+> `docs/architecture/02-deployment-topology.md` §7.3).
+
+### 15.2 The per-run `ExecutionTrace`
+
+Every run carries an `ExecutionTrace` (`core/schemas.py:296-319`) with: `run_id`, `task`, `query`,
+`inputs`, `modalities`, `intent`, `validation`, `workflow`, `steps`, `selected_models`, `parameters`,
+`outputs`, `confidence`, `timings`, `fallbacks`, `errors`, `contradiction`, `config_hash`, `started_at`,
+`finished_at`.
+
+This is the **only** diagnostic surface for a *wrong but successful* answer — no log line records a
+successful run (`docs/architecture/10-observability-and-ops.md` §7.6). The operator reads:
+
+| Trace field | What it answers |
+|---|---|
+| `intent` | what the router thought the query meant |
+| `task` | what was dispatched |
+| `selected_models` | what actually ran |
+| `errors` | what failed inside the run |
+| `fallbacks` | what degraded |
+| `config_hash` | which config produced the result |
+
+### 15.3 The Codespace's own `/v1/health`
+
+The inference tier answers its own health route, derived rather than asserted, with a torch-free device
+probe (`app/space_app.py:521-547`; `core/schemas.py:430-437`; `docs/architecture/10-observability-and-ops.md`
+§2.10). `gpu_available: false` is **expected** on a CPU host and must never be surfaced as a fault
+(`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §5.2).
+
+## 16. What does **not** exist
+
+This is the honest inventory. None of it is aspirational — the list exists so that a reader does not
+assume a monitoring facility that was never built (`docs/architecture/10-observability-and-ops.md` §6):
+
+| Capability | Present? | Evidence |
+|---|---|---|
+| **APM** (application performance monitoring) | **no** | no APM client, SDK or agent in any source read |
+| **Distributed tracing** | **no** | no trace-context propagation; the three id namespaces do not join |
+| **Cost accounting** | **no** | `GPU_DURATIONS` declares durations but nothing meters or reports consumption |
+| **Metrics endpoint** (Prometheus / OpenMetrics) | **no** | no `/metrics` route in any app factory |
+| **Per-model latency histogram** | **no** | `trace.timings` is per-run, per-step, never aggregated |
+| **Error-rate counter** | **no** | no counter exists; `trace.errors` is per-run only |
+| **Request counter** | **no** | the tunnel's `completed` counts only tunnel deliveries, per Render process |
+| **Uptime / restart tracking** | **no** | no uptime field; the hub's counters reset on restart |
+| **Alerting** | **no** | no alerting rule, webhook or threshold anywhere |
+| **Structured / JSON logs** | **no** | all log calls use `%s`-style free text |
+| **Log shipping / aggregation** | **no** | logs are per-host; the Codespace's are on an ephemeral filesystem |
+| **Dashboards** | **no** | none exists |
+| **SLO / SLA definition** | **no** | none exists |
+| **A system-level end-to-end benchmark** | **no** | `DOCS_STYLE_GUIDE.md` §3: *"does not exist; no system-level accuracy is claimed"* |
+
+**There is no pager, no alert, and no dashboard.** An operator learns the system is down by trying to use
+it. This is stated as a fact, not a complaint.
+
+## 17. Why "no alerting" is a design fact, not an oversight
+
+The plan's §74 lists what is **deliberately not included**, and the monitoring gaps are downstream of
+that list:
+
+```
+authentication
+multi-tenant isolation
+distributed queues
+autoscaling
+observability platform
+Kubernetes
+service mesh
+distributed storage
+horizontal worker orchestration
+enterprise security
+billing
+SLA infrastructure
+```
+
+(`Implementation and Architecture plan.md` §74)
+
+The architecture is described there as **scale-compatible, but not a production implementation**. The
+absence of an observability platform, billing and SLA infrastructure is therefore intentional at this
+stage. An operator should not expect — and must not claim — production-grade monitoring.
+
+---
+
+# Part VI — Capacity and cost
+
+## 18. The capacity shape
+
+Three facts bound capacity, and none of them is elastic:
+
+| Property | Value | Evidence |
+|---|---|---|
+| Render plan | **free tier** — sleeps when idle | `render.yaml` (`plan: free`); `release/repo/docs/DEPLOYMENT.md` §8 |
+| Inference hosts | **one** Codespace | `release/CURRENT_RELEASE_STATE.md` §1 |
+| Device | **CPU-only** | `render.yaml`; `.devcontainer/devcontainer.json`; `docs/DEPLOYMENT_DECISION.md` §5 |
+| Autoscaling | **absent** | plan §74 lists `autoscaling` under **Not included** |
+| Horizontal workers | **absent** | plan §74 lists `horizontal worker orchestration` under **Not included** |
+| Database / queue | **absent** | the gateway has *"no database, no auth, no queue"* (`deploy/render/main.py:4-6`) |
+
+The consequence for an operator:
+
+- **A single client can occupy the system.** The per-IP rate limit is a fairness control, **not** a
+  security control (§19.2). There is no queue to absorb a burst.
+- **A restart is a full outage.** There is no replica to fail over to. The Codespace filesystem is
+  ephemeral (`deploy/codespace/launch.sh:44-47`), so a restart also empties the asset store.
+- **Cold starts are unavoidable.** Render's free tier sleeps, so the first request after idle pays the
+  cold-start cost (§10).
+
+## 19. The cost shape, and the one absence that has a code artifact
+
+### 19.1 What is declared
+
+`app/space_app.py` declares a per-task ZeroGPU duration, transcribed from the frozen config
+(`app/space_app.py:105-116`, quoted in `docs/architecture/10-observability-and-ops.md` §6.1):
+
+```python
+GPU_DURATIONS: dict[str, int] = {
+    "vqa": 20,
+    "caption": 20,
+    "grounding": 45,
+    "change": 30,
+    "optical_sar": 45,
+    "change_vqa": 30,
+}
+```
+
+These values are used **only** to decorate a handler with a ZeroGPU reservation. On the live **CPU**
+deployment the decoration is a **no-op**: `_spaces_module()` returns `None` when the `spaces` package is
+absent, so `decorate_gpu` returns the identity decorator (`app/space_app.py:143-166`).
+
+> **Do not present `GPU_DURATIONS` as a cost model.** It is a declaration of intended reservation, and on
+> the CPU deployment it reserves nothing (`docs/architecture/10-observability-and-ops.md` §6.1). The
+> ZeroGPU 5-GPU-minute/day quota and the `@spaces.GPU` decoration are **frozen paperwork** — no Gradio
+> runtime exists in code, and the manifest is left undisturbed because editing it would move the config
+> hash (`release/repo/docs/DEPLOYMENT.md` §11).
+
+### 19.2 What is **not** metered
+
+Nothing reads `GPU_DURATIONS` back out to compute, record or report consumption. There is no per-run
+GPU-second field, no cumulative counter, and no budget-exhaustion signal
+(`docs/architecture/10-observability-and-ops.md` §6.1).
+
+**No cost is observed.** There is no cost accounting for:
+
+| Cost | Metered? | Note |
+|---|---|---|
+| Render compute | **no** | free tier; no usage field is read |
+| Codespace compute | **no** | no usage field is read; the quota is a platform property |
+| Hugging Face model hosting | **no** | no usage field is read |
+| Model download volume | **no** | `warm_cache.py` reports per-model status but not bytes or cost |
+| Per-request inference cost | **no** | no field exists |
+
+The rate-limit and size-limit values are the implementation's choices, not the plan's, and the maintainer
+should confirm them because they bound one client's share of capacity
+(`docs/PHASE19_FINAL_HARDENING.md` §4.4).
+
+### 19.3 The rate limit is fairness, not protection
+
+The limiter keys on the first hop of `X-Forwarded-For`, which is **client-supplied**. A caller that varies
+the header is never throttled. Measured in-process at 3 requests / 60 s, 8 requests sent: `5/8` throttled
+without the header, **`0/8` with a fresh value per request** (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §4.1.2).
+
+> **Do not treat `SATQUERY_RATE_LIMIT_PER_IP` as protecting capacity.** It bounds accidental loops and
+> honest clients. Fixing it correctly depends on how many proxy hops Render inserts, which must be
+> measured on a deployed gateway — hard-coding a guess would replace a documented weakness with an
+> undocumented one (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §4.1.2).
+
+---
+
+# Part VII — Incident triage
+
+## 20. The symptom → cause → check → action table
+
+One table, ordered by how often each symptom is seen. "First check" is the single cheapest command that
+distinguishes the cases.
+
+| Symptom | Likely cause | First check | Action |
+|---|---|---|---|
+| Request hangs ~249 s, then `504` | **B-07 tunnel gap** | re-read `tunnel.agent_connected` | retry; keep the Codespace warm (§13) |
+| Request returns `503` quickly, `recoverable: true` | no agent connected and `mode == tunnel` | `GET /api/health` | start the Codespace (§7) |
+| `GET /api/health` unreachable | Render sleeping or down | re-issue the request | the first request wakes Render; retry |
+| `GET /api/health` OK but `agent_connected: false` | Codespace stopped, or the agent was reaped | — | start the Codespace; confirm `postStartCommand` ran (§7) |
+| `504 wake_timeout` | the Codespace did not come up within 120 s | `tunnel.agent_connected` | the Codespace is not coming up; check it directly |
+| `502 upstream_unreachable` | a connection error to the Codespace | `tunnel.agent_connected` | check the transport |
+| `422 invalid_request` | the body did not match `AnalysisRequest` | the client's request body | fix the client; **not** a server fault |
+| `503 model_unavailable` | an artifact is absent | `GET /api/capabilities` reasons | expected if not uploaded; ship degraded or upload it |
+| `503 model_load_error` | an artifact is present but corrupt | the capability's `reason` | **a defect** — replace the artifact and report |
+| `confidence.method: "uncalibrated"` | the calibration artifact was not uploaded | the run's trace | honest, not broken |
+| `status: "degraded"` on `/v1/health` | at least one capability is not servable | `GET /api/capabilities` | not an error; the service is up |
+| `gpu_available: false` | CPU host | — | expected; never surface as a fault |
+| a handle that worked now returns `400 input_error` | the handle lapsed, or the Codespace restarted and its asset dir is ephemeral | re-upload | do not assume handles persist |
+| a capability `available: true` **with** a non-null `reason` | hub-backed, no local checkpoint — the first call will be slow | the reason string | not a fault; do not surface as an error |
+| a **wrong but successful** answer | a router/dispatch/artifact issue | the run's `ExecutionTrace` | read `intent`, `task`, `selected_models`, `errors`, `fallbacks` (§15.2) |
+
+The first four rows are the ones an operator hits in practice. Rows 8–15 are from
+`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §5.2, which is the canonical degraded-state reference.
+
+## 21. The consolidated decision tree
+
+For the case where something is wrong and the operator has only the client-side report
+(`docs/architecture/10-observability-and-ops.md` §7.6):
+
+```mermaid
+flowchart TD
+    A["something is wrong"] --> B["GET /api/health"]
+    B --> C{"tunnel.agent_connected?"}
+    C -->|"false"| D["warm the stack<br/>§6 steps 1-2"]
+    C -->|"true"| E{"what did the client see?"}
+
+    E -->|"nothing, request hung"| F{"waited ≈249 s?"}
+    F -->|"yes"| G["B-07 tunnel gap<br/>retry + warm<br/>§13"]
+    F -->|"no"| H["still waiting —<br/>within budget"]
+
+    E -->|"a 5xx"| I["read the error code<br/>§22"]
+    E -->|"a 4xx"| J["a client bug —<br/>the body did not match the contract"]
+
+    I --> K{"code present in the<br/>DEPLOYED revision?"}
+    K -->|"no"| L["the patch is not deployed<br/>§13"]
+    K -->|"yes"| M["act on the code"]
+
+    E -->|"a result, but wrong"| N["read trace:<br/>intent, task, selected_models,<br/>errors, fallbacks<br/>§15.2"]
+```
+
+## 22. The machine-code reference
+
+| Code | Status | Meaning | Action |
+|---|---|---|---|
+| `tunnel_offline` | 503 | no agent is connected and `mode == "tunnel"` | start the Codespace |
+| `forward_unavailable` | 503 | the forwarded port is not anonymously reachable | start the tunnel agent (**patch only**) |
+| `wake_timeout` | 504 | the wake loop exhausted `wake_timeout_s` | the Codespace is not coming up |
+| `upstream_timeout` | 504 | the tunnel was healthy but no agent completed in `tunnel_timeout_s` | retry; check the agent (**patch only**) |
+| `upstream_unreachable` | 502 | a connection error to the Codespace | check the transport |
+| `invalid_request` | 422 | the body was not a valid `AnalysisRequest` | fix the client |
+| `model_unavailable` | 502 | the gateway could not reach the analysis service | retry |
+| `orchestrator_config_error` | 500 | missing `GITHUB_TOKEN` or `CODESPACE_NAME` | fix the env vars (`deploy/render/main.py:256-262`) |
+| `upstream_error` | 502 | a non-connection httpx error | check the upstream (`deploy/render/main.py:392-400`) |
+| `schema_validation_error` | 502 | a non-JSON upstream body | check the upstream (`deploy/render/main.py:404-414`) |
+
+(`deploy/render/main.py:247-291,383-414`; the full taxonomy is in
+[08 — The API Contract](architecture/08-api-contract.md) §12.)
+
+> **The two codes marked "patch only" do not exist in the deployed revision.** An operator on the
+> deployed system will not see `forward_unavailable` or `upstream_timeout`; they will see `wake_timeout`
+> after ≈249 s instead (§13).
+
+---
+
+# Part VIII — Routine maintenance
+
+## 23. Rotating credentials (procedure only)
+
+> **This document records no credential value, and no path to a credential file.** The repository's own
+> evidence index records where credentials are held (`release/CURRENT_RELEASE_STATE.md` §7) by
+> **location and kind only**; that index is not reproduced here. The procedure below is deliberately
+> value-free.
+
+Four credential purposes exist, and each is rotated by changing a value the operator holds — never a
+value in this document:
+
+| Purpose | Where it is consumed | What rotation changes |
+|---|---|---|
+| Codespace control (wake) | the Render env var `GITHUB_TOKEN` | the token the orchestrator uses to `GET`/`POST .../start` a Codespace (`deploy/render/codespaces.py:65-70`) |
+| Repository writes | the operator's local tooling | the token used for the Git Data API deploy path (`release/repo/docs/DEPLOYMENT.md` §7.1) |
+| Codespace account access | the GitHub account session | the account credential used to open the Codespace |
+| Hugging Face model access | the Hub download path | the token used to resolve pinned model revisions |
+
+**The procedure, in order:**
+
+1. **Create the replacement credential** in the provider's UI, with the minimum scope the purpose needs.
+   For the wake path the scope is `codespace` (`deploy/render/codespaces.py:50`, `:90-92`).
+2. **Update the consumer's configuration.** For the wake path this is the Render environment variable
+   `GITHUB_TOKEN` (`render.yaml:14-15`; `release/repo/docs/DEPLOYMENT.md` §6.1).
+3. **Redeploy / restart the consumer** so it reads the new value. The orchestrator reads its environment
+   once at process start, so a value change without a restart is invisible.
+4. **Verify.** Call `GET /api/health` and confirm `config.has_github_token: true`
+   (`deploy/render/main.py:454`). Then trigger one wake to confirm the token is accepted end to end.
+5. **Revoke the old credential** at the provider, once step 4 passes.
+
+**Two rules that must not be broken:**
+
+- **Never set a token variable to an empty string.** An empty `HF_TOKEN` produced
+  `Authorization: Bearer `, which httpx rejects with a `LocalProtocolError` that is misreported as an
+  upstream failure. **Omit the variable entirely rather than setting it empty**
+  (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §4.1). The same reasoning applies to `GITHUB_TOKEN`: the
+  orchestrator raises a clear `orchestrator_config_error` when it is absent
+  (`deploy/render/main.py:90-96`), which is better than a confusing transport failure.
+- **Never commit a credential.** The Render variables are `sync: false` in `render.yaml` and must be set
+  in the dashboard (`deploy/render/README.md` §Environment variables).
+
+> **`has_github_token` is a boolean, never the value.** The health payload reports only whether a token is
+> present (`deploy/render/main.py:454`), so a health probe is a safe way to confirm rotation without
+> exposing the credential.
+
+## 24. Restarting the tunnel agent
+
+The agent is supervised and should not need a manual restart, but the procedure is:
+
+1. **Check whether it is running** — the Codespace-side check is
+   `pgrep -f "deploy/codespace/tunnel_agent.py"` (`deploy/codespace/launch.sh:153,179`).
+2. **Re-run the launcher** — `bash deploy/codespace/launch.sh`. It is **guarded** on the process, so
+   re-running is a no-op if the agent is already up (`deploy/codespace/launch.sh:24,153-155`).
+3. **Confirm the reconnect** — read the agent log for the announce line
+   (`grep -q "announced to hub" "$TUNNEL_LOG"`, `deploy/codespace/launch.sh:185`) and then confirm
+   `tunnel.agent_connected: true` from outside (§6 step 1).
+
+> **The launcher restarts the serve process too, if it is stale** (`deploy/codespace/launch.sh:100-147`).
+> That is intentional: a stale server answering from old code is worse than a restart (§7).
+
+> **Caveat.** The agent log path is `/tmp/satquery-tunnel.log` and the serve log is
+> `/tmp/satquery-serve.log` (`deploy/codespace/launch.sh:63-64`). Both are on the **ephemeral** Codespace
+> filesystem, so they vanish with the Codespace (`deploy/codespace/launch.sh:44-47`).
+
+## 25. Re-warming the model cache
+
+Re-warm after a Codespace rebuild, or whenever the first analysis is slower than expected:
+
+```bash
+python deploy/codespace/warm_cache.py
+```
+
+`post_create.sh` runs this automatically on container **creation** (`deploy/codespace/post_create.sh:5-6`),
+and it is wired as the devcontainer `postCreateCommand` (`.devcontainer/devcontainer.json:17`).
+
+> **Creation vs. start.** `postCreateCommand` runs only when the container is **created**, whereas
+> `postStartCommand` runs on every start (`.devcontainer/devcontainer.json:17-18`). The `containerEnv`
+> block is likewise applied only at creation, which is why `launch.sh` re-exports the asset-upload
+> variables on every start — *"`containerEnv` is only applied when the container is CREATED"*
+> (`deploy/codespace/launch.sh:49-52`). An operator who changes an env var must restart, not just reload.
+
+## 26. Deploying a change (the Git Data API path)
+
+Deployment does **not** use `git push`. Every deployed file is uploaded as a **blob** whose sha256 is
+computed locally and verified against the uploaded blob, then assembled into a tree, committed, and the
+branch ref patched (`release/repo/docs/DEPLOYMENT.md` §7.1).
+
+Why this matters operationally:
+
+- each file is **content-verified** rather than trusted;
+- deletions are expressed explicitly as `sha: null` tree entries;
+- the deploy is **idempotent** — re-running it with identical content produces no change.
+
+**Measured:** 9 deployed files were re-read from the API and found **sha256 byte-identical** to the local
+copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`,
+`release/CURRENT_RELEASE_STATE.md` §5).
+
+**Frontend deploy** (Cloudflare Pages) uses a staging step, not the Git Data API:
+
+```bash
+node scripts/stage_pages.mjs \
+  --out=.deploy/dist-final \
+  --include=_headers \
+  --include=robots.txt \
+  --include=assets/img/eo/provenance.json \
+  --include=assets/img/eo/CREDITS.md
+
+npx wrangler pages deploy "C:/Users/anish/satquery-ai/.deploy/dist-final" --project-name <name>
+```
+
+(`docs/DEPLOYMENT_DECISION.md` §7)
+
+`_headers` and `robots.txt` must be **force-included** because no page references them; `provenance.json`
+and `CREDITS.md` likewise (`docs/DEPLOYMENT_DECISION.md` §7).
+
+---
+
+# Part IX — Known operational gaps
+
+## 27. The explicit gaps list
+
+Each row is something an operator might reasonably expect and that does **not** exist. None is a
+regression; each is a boundary of the current release.
+
+| # | Gap | Consequence | Status |
+|---|---|---|---|
+| 1 | **No alerting** of any kind | an outage is discovered by trying to use the system | **not implemented** (§16) |
+| 2 | **No APM / metrics / distributed tracing** | no latency, error-rate or throughput trend exists | **not implemented** |
+| 3 | **No cost accounting** | consumption is unmeasured | **not implemented** (§19.2) |
+| 4 | **No dashboard / SLO / SLA** | no shared view of health; no target defined | **not implemented** |
+| 5 | **No structured logs / log shipping** | logs are per-host free text; the Codespace's are ephemeral | **not implemented** |
+| 6 | **B-07 is unfixed in production** | a request can hang ≈249 s then `504` | **`OPEN`** (§13) |
+| 7 | **B-02 trailing `\n`** | cosmetic; a wrong-looking field in health | **`OPEN` (cosmetic)** (§14) |
+| 8 | **No autoscaling, no replicas** | a restart is a full outage | **BY DESIGN** (plan §74) |
+| 9 | **No database, queue or persistence** | no run history survives a restart | **BY DESIGN** (`deploy/render/main.py:4-6`) |
+| 10 | **One Codespace** | capacity is bounded by one CPU host | **BY DESIGN** (§18) |
+| 11 | **No auth** | the contract documents *"no auth in v1"*; paths are scrubbed from client-visible fields as a partial mitigation | **BY DESIGN** |
+| 12 | **A system-level E2E benchmark does not exist** | no single system accuracy number can be quoted | **NOT RUN** |
+| 13 | **No measured cold-start distribution** | only *"tens of seconds"* is documented | **`UNKNOWN`** |
+| 14 | **The effective log level / destination per host** | not established | **`UNKNOWN`** |
+| 15 | **Whether `doctor.sh` / `tunnel_agent.py` exist in the deployed repo** | the monorepo copy is stale and lacks them | **`UNKNOWN`** (§7) |
+| 16 | **The B-07 patch is not deployed** | the fast-fail codes are absent in production | **`OPEN`** (§13) |
+
+## 28. `NOT RUN` / `OPEN` / `BLOCKED` / `UNKNOWN` for operations
+
+| # | Item | Status |
+|---|---|---|
+| 1 | B-07 — tunnel gaps; patch prepared, **not deployed** | **`OPEN`** |
+| 2 | B-02 — `/api/health` `codespace_name` trailing `\n` | **`OPEN` (cosmetic)** |
+| 3 | A deployed system-level load test | **`NOT RUN`** |
+| 4 | A measured cold-start distribution | **`NOT RUN`** — only "tens of seconds" is documented |
+| 5 | Multi-region / HA deployment | **`NOT RUN`** |
+| 6 | A system-level end-to-end benchmark | **`NOT RUN`** — none exists |
+| 7 | Any APM / metrics / distributed tracing / alerting | **not implemented** |
+| 8 | Cost accounting | **not implemented** — `GPU_DURATIONS` is declared, not metered |
+| 9 | The effective log level and destination on each host | **`UNKNOWN`** |
+| 10 | Whether the deployed `SatQuery-Inference` repo carries `doctor.sh` / `tunnel_agent.py` | **`UNKNOWN`** |
+| 11 | Log retention on Render | **`UNKNOWN`** — a platform property, not observable from the code |
+| 12 | The ZeroGPU/Gradio deployment target | **`REJECTED`** (superseded; frozen paperwork only) |
+| 13 | The five historical backend blockers | **closed by construction, not proven in production** |
+
+> Rows 9 and 11 are copied from `docs/architecture/10-observability-and-ops.md` §8.1 so that the two
+> documents cannot drift. Row 13 is the honest framing from `release/repo/docs/DEPLOYMENT.md` §9: the
+> design closes the blockers, and the first live run is what would *verify* them.
+
+---
+
+# Part X — Evidence
+
+## 29. Where the evidence lives
+
+| What | Where |
+|---|---|
+| the live health payload | `release/CURRENT_RELEASE_STATE.md` §1; `release/repo/docs/DEPLOYMENT.md` §5 |
+| the live capability contract | `release/CURRENT_RELEASE_STATE.md` §1 |
+| the deployed revisions | `release/CURRENT_RELEASE_STATE.md` §1; `release/repo/docs/DEPLOYMENT.md` §1 |
+| the B-07 root shape and ≈249 s | `release/CURRENT_RELEASE_STATE.md` §6; `release/repo/docs/DEPLOYMENT.md` §8.1 |
+| the undeployed B-07 patch | session scratch: `fix-b07-forward-unavailable.patch` |
+| B-02's status and witness role | `release/repo/docs/DEPLOYMENT.md` §5; `docs/architecture/10-observability-and-ops.md` §7.4 |
+| the cold-start shape and the wake loop | `deploy/render/main.py:76-78,299-356,486-489` |
+| the Codespace launcher | `deploy/codespace/launch.sh`; `.devcontainer/devcontainer.json:18` |
+| the warm-up contract | `deploy/codespace/warm_cache.py` |
+| the asset-upload environment | `deploy/codespace/launch.sh:36-54` |
+| the deploy mechanics (Git Data API) | `release/repo/docs/DEPLOYMENT.md` §7 |
+| the platform traps | `release/repo/docs/DEPLOYMENT.md` §10; `release/repo/docs/REPRODUCIBILITY.md` §10 |
+| the historical backend blockers | `docs/DEPLOYMENT_DECISION.md` §8; `release/repo/docs/DEPLOYMENT.md` §9 |
+| the frozen config and hash | `configs/base.yaml`; `core/config.py:76-80` |
+| the observability inventory | `docs/architecture/10-observability-and-ops.md` §6 |
+| the triage tables | `docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §5.2, §7 |
+| the live validation (3 passes, 24 runs) | `.workbuddy-ai/scratch/live_validation/` |
+| the rate-limit finding (F-5) | `docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §4.1.2 |
+
+### Cross-references
+
+| For | See |
+|---|---|
+| the four tiers, the tunnel, the wake flow, `transport_mode` | [02 — Deployment Topology](architecture/02-deployment-topology.md) |
+| health, counters, traces, what is and is not observed | [10 — Observability and Operations](architecture/10-observability-and-ops.md) |
+| the four endpoints, the envelopes, the error taxonomy | [08 — The API Contract](architecture/08-api-contract.md) |
+| the request lifecycle and the nine-state spine in motion | [03 — Request Lifecycle](architecture/03-request-lifecycle.md) |
+| the frozen config and `Config.hash == 78f1e3700da15aa1` | [07 — Configuration and Freeze](architecture/07-configuration-freeze.md) |
+| live revisions, env vars, deploy mechanics, platform traps | [../DEPLOYMENT.md](DEPLOYMENT.md) |
+| what a third party can and cannot reproduce | [../REPRODUCIBILITY.md](REPRODUCIBILITY.md) |
+| how to build, test and extend the codebase | [../DEVELOPMENT.md](DEVELOPMENT.md) |
+
+---
+
+> **Chapter summary.** SatQuery AI runs four tiers — a static frontend, a thin Render orchestrator, one
+> CPU Codespace reached over an outbound tunnel, and a Hugging Face model tier — with exactly one
+> inference host and no replicas. The operator's single most important check is
+> `GET /api/health` → `tunnel.agent_connected: true`; the single most important diagnostic is the
+> ≈249 s-then-`504` signature of a B-07 tunnel gap. **B-07 is `OPEN` and the patch is not deployed; B-02
+> is `OPEN` and cosmetic.** The system observes three things — the transport, one run's trace, and the
+> server logs — and has **no** alerting, APM, distributed tracing or cost accounting. Capacity is one
+> free-tier Render service and one CPU Codespace; nothing is metered. Four things are genuinely
+> `UNKNOWN — not established from the available evidence`: the effective log level and destination per
+> host, log retention on Render, whether the deployed inference repository carries the tools the monorepo
+> launcher references, and any measured cold-start distribution.