thundercode commited on
Commit
af0705e
·
verified ·
1 Parent(s): 5a57728

release: add docs/DEPLOYMENT.md

Browse files
Files changed (1) hide show
  1. docs/DEPLOYMENT.md +544 -64
docs/DEPLOYMENT.md CHANGED
@@ -3,18 +3,69 @@
3
  **Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `OPEN` · `DEFERRED` · `BLOCKED` · `BY DESIGN`.
4
 
5
  The live SatQuery AI system runs across **three private repositories** plus one **public umbrella
6
- repository**. The monorepo working copy is **not** the deployed source. This document records the live
7
- topology, every environment variable, the deploy mechanics, the cold-start behaviour, the known
8
- issues, and the platform traps that cost real debugging time.
 
 
 
 
 
 
9
 
10
  > **The single most important trap in this document.** `deploy/` inside the monorepo is **stale and
11
- > untracked**. It is **not** the deployed source. Edits must go to the three real repositories.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
  ---
14
 
15
- ## 1. Live revisions (VERIFIED)
16
 
17
- Read from the GitHub API during the release reconnaissance.
18
 
19
  | Component | Repository | Visibility | Branch | Revision | Host |
20
  |---|---|---|---|---|---|
@@ -23,11 +74,32 @@ Read from the GitHub API during the release reconnaissance.
23
  | Inference | `Anish-lab-blip/SatQuery-Inference` | **private** | `main` | **`5a0936ace491`** | Codespace `potential-space-trout-r4ppw969w45j2pvvw`, port 8000 |
24
  | Public umbrella | `Anish-lab-blip/SatQuery-AI` | **public** | `main` | `3dcabd32da41` ("Initial commit") | this release home |
25
  | Monorepo (working copy) | `C:/Users/anish/satquery-ai` | local only | `master` | `9d57aed` | **no git remote**; 334 dirty entries |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
 
27
- The public umbrella repository previously contained **only a 13-byte README** (`# SatQuery-AI`). It is
28
- the intended home for this release.
29
 
30
- ## 2. The four tiers
31
 
32
  ```
33
  Browser
@@ -58,62 +130,207 @@ flowchart LR
58
  R -->|"envelope + error translation"| CF
59
  ```
60
 
61
- ## 3. Why each tier exists
 
62
 
63
  ### 3.1 Cloudflare Pages — the static tier
64
 
65
  Serves the frontend. **No backend, no secrets, and no API calls of its own** on the static pages. The
66
- one exception is the Analyze console, which calls the gateway. Staged by `scripts/stage_pages.mjs` and
67
- deployed with `npx wrangler pages deploy`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
 
69
  ### 3.2 Render — the gateway
70
 
71
- A deliberately **thin, stateless** orchestrator. Its responsibilities, from
72
- `docs/DEPLOYMENT_ARCHITECTURE.md` §2:
 
73
 
74
  | Responsibility | Detail |
75
  |---|---|
76
  | Schema validation | reject malformed requests before they cost inference |
77
  | Size limits | whole-request body cap, shared with the Codespace |
78
- | Rate limiting | per-IP count + window — **fairness, NOT a security control** |
79
  | CORS allowlist | the Pages origin; **never `*`** |
80
  | Request ids | correlate a request across tiers |
81
- | Timeouts | sit inside the task budget |
82
  | Secret custody | holds credentials that must never reach the browser |
83
- | Error translation | upstream failures → the documented error envelope |
84
 
85
  It is **not** a model host. It has **no database, no auth, and no queue**.
86
 
 
 
 
 
 
 
 
 
 
87
  > **Hard rule.** The gateway must **not** retry `POST /api/infer` on its own — a retry would consume
88
- > inference a second time. The client decides on retry.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
89
 
90
  ### 3.3 GitHub Codespace — the inference tier
91
 
92
  Runs the real inference service: `build_space_app()` from `app/space_app.py`, served by
93
  `deploy/codespace/serve.py` on `$PORT`, in **CPU mode**. It honours the four-endpoint contract
94
  (`/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets`), imports cheaply without torch, reuses
95
- `app/serving.py` as the composition root, and **degrades rather than crashes** on absent artifacts.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
96
 
97
  ### 3.4 Hugging Face — the model tier
98
 
99
  Holds the six trained artifacts and the model card. It is **not** the runtime inference host; the
100
- Codespace resolves the pinned backbones from the Hub at run time.
 
 
 
 
101
 
102
  ## 4. Why the transport is an outbound tunnel
103
 
104
  The inference host is a Codespace in a **private** repository. A forwarded port for a private repo
105
  returns **`302`**, so an inbound-forwarding design cannot work. Instead:
106
 
107
- - the Codespace runs `deploy/codespace/tunnel_agent.py`;
108
  - the agent **dials out** to `POST /tunnel/agent` and long-polls;
109
  - work is executed against `http://127.0.0.1:8000` **locally**.
110
 
111
- This inverts the usual direction: the inference host needs **no inbound firewall hole**. It also means
112
- the transport is only alive while the agent is polling.
 
113
 
114
  **Measured:** `GET /api/health` reported `tunnel.agent_connected: true` with a non-zero `completed`
115
  counter, and `POST /api/infer {}` returned `422 invalid_request` with the response header
116
- `x-satquery-transport: tunnel`.
 
 
 
 
 
 
 
 
 
 
 
 
117
 
118
  ## 5. The full live health payload (VERIFIED, probed)
119
 
@@ -140,8 +357,39 @@ counter, and `POST /api/infer {}` returned `422 invalid_request` with the respon
140
  }
141
  ```
142
 
143
- Note `codespace_name` still carries a **trailing `\n`** — this is **B-02**, cosmetic and `OPEN`; the
144
- wake path strips it (`_codespace_name()`), so only the health payload reports the raw value.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
145
 
146
  ## 6. Environment variables
147
 
@@ -153,30 +401,56 @@ wake path strips it (`_codespace_name()`), so only the health payload reports th
153
  | `CODESPACE_PORT` | `8000` | the inference port |
154
  | `SATQUERY_ALLOWED_ORIGINS` | `https://satquery.pages.dev` | CORS allowlist (never `*`) |
155
  | `SATQUERY_DEVICE` | `cpu` | device preference |
156
- | `SATQUERY_TRANSPORT` | `auto` | tunnel first, then forward |
157
  | `SATQUERY_TUNNEL_TIMEOUT_S` | `150` | how long to wait on the tunnel |
158
  | `SATQUERY_WAKE_TIMEOUT_S` | `120` | how long to wait for a cold start |
159
  | `SATQUERY_UPSTREAM_TIMEOUT_S` | `90` | gateway → upstream budget |
160
- | `GITHUB_TOKEN` | present | Codespace control |
161
  | `PORT` | platform-supplied | Render's own listen port |
162
 
 
 
 
 
 
 
 
 
 
 
163
  > **Measured absence.** There is **no** `SATQUERY_UPSTREAM_URL` and **no** `HF_TOKEN` in the live
164
  > config. The transport is the outbound tunnel, not a forwarded port. This contradicts the older
165
- > `docs/DEPLOYMENT_TOPOLOGY.md` §3.2 table, which predates the tunnel design.
 
166
 
167
  ### 6.2 Codespace (inference)
168
 
169
  | Variable | Purpose |
170
  |---|---|
171
- | `PORT` | platform-assigned; **must be read** (a historical blocker was that no code read it) |
172
  | `SATQUERY_DEVICE` | `cpu` \| `cuda` \| `mps` \| `null`; read **without importing torch** |
173
- | `SATQUERY_MAX_FILE_BYTES` | per-file cap, shared with Render |
174
  | `SATQUERY_ASSET_ENABLED` / `SATQUERY_ASSET_DIR` | both required for `/v1/assets`; **fails closed (503)** otherwise |
175
  | `SATQUERY_ASSET_MAX_FILES` / `SATQUERY_ASSET_TTL_S` | optional handle capacity / lifetime |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
176
 
177
  ### 6.3 Config-loader environment overrides
178
 
179
- Two registry values can be overridden from the environment without editing the YAML
180
  (`core/config.py`):
181
 
182
  | Variable | Effect |
@@ -184,10 +458,22 @@ Two registry values can be overridden from the environment without editing the Y
184
  | `SATQUERY_PRECISION` | overrides `training.precision` |
185
  | `SATQUERY_TORCH_COMPILE` | overrides `deployment.torch_compile` (`"true"` → `True`) |
186
 
187
- Both are still validated by the loader — setting `SATQUERY_TORCH_COMPILE=true` **fails startup**
188
- because finding C-8 forbids `torch.compile`.
 
 
 
 
 
189
 
190
- ## 7. Deploy mechanics
 
 
 
 
 
 
 
191
 
192
  | Tier | Mechanism |
193
  |---|---|
@@ -196,17 +482,98 @@ because finding C-8 forbids `torch.compile`.
196
  | Inference → Codespace | `deploy/codespace/serve.py` binds `build_space_app()` to `$PORT`; `.devcontainer/` forwards 8000 and starts the tunnel agent via `postStartCommand` |
197
  | Repository writes | the **GitHub Git Data API** — blob → tree → commit → `PATCH` ref |
198
 
199
- ### 7.1 Why the Git Data API instead of `git push`
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
200
 
201
  Every deployed file is uploaded as a **blob** whose sha256 is **computed locally and verified against
202
- the uploaded blob**, then assembled into a tree, committed, and the branch ref patched. This means:
 
203
 
204
  - each file is **content-verified** rather than trusted;
205
- - deletions are expressed explicitly as `sha: null` tree entries;
206
  - the deploy is **idempotent** — re-running it with identical content produces no change.
207
 
208
  **Measured:** 9 deployed files were re-read from the API and found **sha256 byte-identical** to the
209
- local copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
210
 
211
  ## 8. Cold start (documented, not hidden)
212
 
@@ -220,6 +587,9 @@ shows *"Waking inference engine…"* during this.
220
  | Tunnel wait before falling through | `SATQUERY_TUNNEL_TIMEOUT_S` = 150 s |
221
  | Wake wait | `SATQUERY_WAKE_TIMEOUT_S` = 120 s |
222
  | Upstream budget | `SATQUERY_UPSTREAM_TIMEOUT_S` = 90 s |
 
 
 
223
 
224
  Cold start is **documented rather than papered over**: an honest "this will take a while the first
225
  time" is better than a silent hang.
@@ -236,48 +606,98 @@ The forward path to a **private** repo returns `302` quickly — but the wake st
236
  ```
237
 
238
  This is the **root shape** of the observed transient tunnel gap, and it is why a request can appear to
239
- hang and then fail. A patch (`fix-b07-forward-unavailable.patch`) was authored and verified
240
- (`py_compile` clean, applies cleanly to the deployed `main.py`), adding distinct
241
- `forward_unavailable` (503, terminal 302/401/403 on the forward path) and `upstream_timeout`
242
- (504, tunnel healthy but slow) codes, plus the `codespace_name` `.strip()` fix.
 
 
 
 
 
 
243
 
244
  > **Status: B-07 is `OPEN`.** The patch is **prepared but NOT deployed.** The deployed health payload
245
- > still shows the trailing `\n`.
 
 
 
 
 
 
 
 
 
 
 
 
246
 
247
  ## 9. The five historical backend blockers
248
 
249
- Before any backend could boot, five verified blockers had to be closed. The current design closes them:
 
250
 
251
  | # | Blocker (verified) | How it is closed |
252
  |---|---|---|
253
  | 1 | `requirements.txt` declared no `fastapi` / `uvicorn` / `httpx` / `starlette` | the runtime installs the ASGI stack so `build_space_app()` and the gateway can import |
254
  | 2 | no code read `$PORT` — a platform port would be ignored | `deploy/codespace/serve.py` binds to `$PORT`; Render reads its own |
255
- | 3 | hand-rolled CORS raised `405` on `OPTIONS`, so browser preflight failed | the gateway registers `OPTIONS` explicitly / uses Starlette's CORS middleware |
256
  | 4 | module-level `app = create_app()` swallowed config errors into `app = None` | construction errors now **propagate** (fail-fast) instead of leaving a dead `app` |
257
  | 5 | adapter integrity unverified on load (`_adapter_sha256` computed but never compared) | the load path compares the digest against an expected value, or fails startup |
258
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
259
  ## 10. Platform traps (recorded so they are not rediscovered)
260
 
261
  | Trap | Detail |
262
  |---|---|
263
- | **Cloudflare `_headers` CONCATENATE** | Two matching rules are **merged, not overridden**. A specific rule nested under a broad `/assets/img/*` rule yields `max-age=604800, …, max-age=0, must-revalidate` — and Chromium takes the **FIRST** `max-age`. The file's own "later rules override" comment is **false**. |
264
  | **Cloudflare 308 redirect** | `X.html` → `/X`. Reference the extensionless path. |
265
- | **Forwarded port returns 302** | for a private repo — this is *why* the tunnel exists. |
266
  | **Tunnel agent must start on boot** | via the devcontainer `postStartCommand`, or a restarted Codespace comes up with `agent_connected: false`. |
267
- | **Never retry `/api/infer` at the gateway** | a retry consumes inference twice. |
268
- | **`deploy/` is stale and untracked** | not the deployed source. |
 
 
 
 
 
269
 
270
  ## 11. The superseded design, and what did NOT change
271
 
272
  The earlier design ran inference on an **HF Space with ZeroGPU** (5 GPU-min/day, `@spaces.GPU`
273
- decoration) behind a **Railway** gateway. The active design changes three things:
 
274
 
275
  1. **CPU-first instead of ZeroGPU.** No code change was required — `device_preference` honours
276
  `SATQUERY_DEVICE` and defaults to CPU, every specialist defaults to `device="cpu"`, and all
277
- placement is `.to(device)` (never `.cuda()`).
 
278
  2. **A real, always-buildable inference environment.** A Codespace gives a reproducible container
279
- without a GPU quota or a Space's cold-start constraint. The wake flow replaces ZeroGPU lazy
280
- loading as the cold-start story.
281
  3. **No GPU quota to protect at the gateway.** Rate/size limits remain, but as **fairness** controls
282
  rather than quota protection.
283
 
@@ -287,39 +707,99 @@ decoration) behind a **Railway** gateway. The active design changes three things
287
  |---|---|
288
  | the four-endpoint contract | `/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets` |
289
  | the gateway responsibility table | §3.2 above |
290
- | the env-var vocabulary | only host names moved (`SATQUERY_SPACE_URL` → `SATQUERY_UPSTREAM_URL`) |
291
  | the config freeze | `78f1e3700da15aa1` |
292
 
293
  `configs/deploy.yaml` still describes the old HF-Space/ZeroGPU target (`platform: huggingface-spaces`,
294
  `sdk: gradio`, `zerogpu: true`, the `gpu_duration_*` values). It is **frozen paperwork**: no Gradio
295
- runtime exists in code, and editing it would move `Config.hash`. It is left undisturbed.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
296
 
297
  ## 12. Deployment preconditions (open items)
298
 
299
- 1. Cloudflare Pages project name / domain (needed for the deploy command and `robots.txt` sitemap).
300
- 2. Artifacts present, **or** capabilities honestly ship `available: false` — the service degrades
 
 
301
  rather than crashes on absent artifacts.
302
- 3. `HF_TOKEN` on Render **if** the HF proxy path is used (not used in the live config).
303
- 4. Codespace `.devcontainer/` forwarding `:8000` **and** starting the tunnel agent.
304
- 5. The five blockers in §9 closed and verified on the first live run.
 
 
 
 
 
 
305
 
306
  ## 13. What is `NOT RUN` / `OPEN` / `BLOCKED` for this topic
307
 
308
  | Item | State |
309
  |---|---|
310
- | B-07 tunnel fallthrough | **OPEN** — patch prepared, NOT deployed |
311
  | B-02 `codespace_name` trailing `\n` | **OPEN** (cosmetic) |
312
  | A deployed system-level load test | **NOT RUN** |
313
  | Multi-region / HA deployment | **NOT RUN** |
314
  | A production database / persistence | **BY DESIGN** (absent) |
 
315
  | The ZeroGPU/Gradio deployment target | **REJECTED** (superseded; frozen paperwork only) |
 
 
 
 
316
 
317
  ## 14. Where the evidence lives
318
 
319
  | Evidence | Location |
320
  |---|---|
321
  | Live health + capabilities probe | `release/CURRENT_RELEASE_STATE.md` §1 |
322
- | Topology contract | `docs/DEPLOYMENT_TOPOLOGY.md`, `docs/DEPLOYMENT_ARCHITECTURE.md` |
 
 
 
 
 
323
  | The undeployed B-07 patch | session scratch: `fix-b07-forward-unavailable.patch` |
324
  | Deployed-artifact integrity check | `verify_deployed_head.py` (session scratch) |
325
- | Live validation (3 passes) | `.workbuddy-ai/scratch/live_validation/` |
 
 
3
  **Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `OPEN` · `DEFERRED` · `BLOCKED` · `BY DESIGN`.
4
 
5
  The live SatQuery AI system runs across **three private repositories** plus one **public umbrella
6
+ repository**, serving a **static frontend** on Cloudflare Pages, a **thin orchestrator / gateway** on
7
+ Render, and a **CPU inference service** in a GitHub Codespace reached over an **outbound tunnel**. The
8
+ monorepo working copy is **not** the deployed source.
9
+
10
+ This document is the exhaustive deployment reference: the four tiers, every live revision, every
11
+ environment variable (with measured live values), the deploy mechanics per tier, cold-start
12
+ behaviour, the five historical backend blockers, the platform traps, and the superseded design that
13
+ the active one replaced. It is written to be readable *without* the source tree, but every non-obvious
14
+ claim carries the file it came from.
15
 
16
  > **The single most important trap in this document.** `deploy/` inside the monorepo is **stale and
17
+ > untracked**. It is **not** the deployed source. The deployed backend is `SatQuery-Backend/main.py`;
18
+ > the monorepo's `deploy/render/main.py` is an earlier, tunnel-less revision. Edits must go to the
19
+ > three real repositories, never to the local `deploy/` tree.
20
+
21
+ **Companions.** [`architecture/02-deployment-topology.md`](architecture/02-deployment-topology.md)
22
+ (the long-form topology treatment), [`OPERATIONS.md`](OPERATIONS.md) (running the live system),
23
+ [`SECURITY.md`](SECURITY.md) (the trust boundary),
24
+ [`architecture/10-observability-and-ops.md`](architecture/10-observability-and-ops.md) (health,
25
+ traces and the operator surface), [`LIMITATIONS.md`](LIMITATIONS.md) §4 (operational limitations),
26
+ [`RESEARCH_NOTES.md`](RESEARCH_NOTES.md) §6 (the `auto`-mode fallthrough), and
27
+ [`TESTING.md`](TESTING.md) (how the deploy-time properties are tested).
28
+
29
+ ---
30
+
31
+ ## 1. How to read this document
32
+
33
+ | Convention | Meaning |
34
+ |---|---|
35
+ | **VERIFIED** | Read from a live endpoint, a git ref, or a file on disk during the release reconnaissance |
36
+ | **MEASURED** | A value with a recorded number and its source |
37
+ | **IMPLEMENTED** | Code exists; whether it ran is stated separately |
38
+ | **NOT RUN** | The work has not been executed |
39
+ | **OPEN** | A known defect or gap that is not closed |
40
+ | **BY DESIGN** | Deliberately absent, and the reason is recorded |
41
+ | **UNKNOWN** | `UNKNOWN — not established from the available evidence` |
42
+
43
+ Every revision, environment variable and finding below was read from a file or a live endpoint. Where
44
+ a value could not be established, the text says so rather than guessing.
45
+
46
+ ### 1.1 The three deploy sources versus the working copy
47
+
48
+ | Artifact | Location | Role |
49
+ |---|---|---|
50
+ | Frontend source | `Anish-lab-blip/SatQuery-Frontend` (private) | staged from the monorepo's `frontend/` contents |
51
+ | Backend source | `Anish-lab-blip/SatQuery-Backend` (private) | Render orchestrator (`main.py`, tunnel client) |
52
+ | Inference source | `Anish-lab-blip/SatQuery-Inference` (private) | Codespace FastAPI + `deploy/codespace/tunnel_agent.py` |
53
+ | Public umbrella | `Anish-lab-blip/SatQuery-AI` (public) | the intended release home |
54
+ | Working copy | `C:/Users/anish/satquery-ai` | **local only, no git remote** |
55
+
56
+ The working copy's `deploy/` is untracked (`git ls-files deploy/` returns empty) and stale. Its
57
+ `deploy/render/main.py` is a **tunnel-less** revision (532 lines by the project's own record); the
58
+ deployed `SatQuery-Backend/main.py` is **768 lines** and carries the tunnel client
59
+ (`docs/FINAL_DELIVERY_TODO.md` §1.1). The tunnel agent
60
+ (`deploy/codespace/tunnel_agent.py`) is **not present in the monorepo working copy at all** — it lives
61
+ in the `SatQuery-Inference` repository, and the monorepo's `deploy/codespace/launch.sh` refers to it as
62
+ a path that only exists in the deployed checkout.
63
 
64
  ---
65
 
66
+ ## 2. Live revisions (VERIFIED)
67
 
68
+ Read from the GitHub API during the release reconnaissance (`release/CURRENT_RELEASE_STATE.md` §1).
69
 
70
  | Component | Repository | Visibility | Branch | Revision | Host |
71
  |---|---|---|---|---|---|
 
74
  | Inference | `Anish-lab-blip/SatQuery-Inference` | **private** | `main` | **`5a0936ace491`** | Codespace `potential-space-trout-r4ppw969w45j2pvvw`, port 8000 |
75
  | Public umbrella | `Anish-lab-blip/SatQuery-AI` | **public** | `main` | `3dcabd32da41` ("Initial commit") | this release home |
76
  | Monorepo (working copy) | `C:/Users/anish/satquery-ai` | local only | `master` | `9d57aed` | **no git remote**; 334 dirty entries |
77
+ | Hugging Face | `thundercode/SatQuery` | **public** | `main` | lastModified `2026-09-25T16:26:53Z` | 2 files only: `.gitattributes`, 25-byte `README.md` |
78
+
79
+ Notes that must not be smoothed over:
80
+
81
+ - The public umbrella `SatQuery-AI` contains **only** `README.md` (13 bytes: `# SatQuery-AI`). At the
82
+ time of the reconnaissance it was effectively empty; it is the intended home for this release.
83
+ - The dirty-entry count is a **snapshot**. `release/CURRENT_RELEASE_STATE.md` §1 records **334** dirty
84
+ entries at release reconnaissance; `docs/FINAL_DELIVERY_TODO.md` §1.1 records **323** (294 untracked,
85
+ 20 modified, 8 deleted) at an earlier capture. The working tree changed between the two captures.
86
+ - The three deployed repositories are **private**. Their GitHub links return `404` for an outside
87
+ audience. This is **BY DESIGN** (see §11.4).
88
+ - The Hugging Face repository `thundercode/SatQuery` carried **only two files** at reconnaissance
89
+ (`.gitattributes` and a 25-byte `README.md`). It is **not** the runtime inference host; the Codespace
90
+ resolves pinned backbones from the Hub at run time.
91
+
92
+ ### 2.1 A note on the "HEAD re-read" verification
93
+
94
+ Nine deployed frontend files were re-read from the GitHub API and found **sha256 byte-identical** to
95
+ the local copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`,
96
+ `release/CURRENT_RELEASE_STATE.md` §5). Three live-validation passes ran against successive HEADs:
97
+ pass 1 against `ff46eba42b18` + `d413d3672311`, passes 2 and 3 against the final HEAD `2d7ae53b482d`.
98
+ No run id is shared between passes.
99
 
100
+ ---
 
101
 
102
+ ## 3. The four tiers
103
 
104
  ```
105
  Browser
 
130
  R -->|"envelope + error translation"| CF
131
  ```
132
 
133
+ The important inversion: the middle arrow is **outbound from the inference host**, not inbound to it.
134
+ That is the whole reason the design works for a private repository (§4).
135
 
136
  ### 3.1 Cloudflare Pages — the static tier
137
 
138
  Serves the frontend. **No backend, no secrets, and no API calls of its own** on the static pages. The
139
+ one exception is the Analyze console (`mission.html`), which calls the gateway.
140
+
141
+ - **Staged by:** `scripts/stage_pages.mjs` (builds a Pages bundle).
142
+ - **Deployed with:** `npx wrangler pages deploy`.
143
+ - **Deploy result (measured, `docs/DEPLOYMENT_DECISION.md` §7):** 60 files staged, 39,173,936 B
144
+ (37.36 MiB) total, largest file `assets/video/satquery-launch-50s.mp4` at 22,710,313 B (21.66 MiB),
145
+ with 3,504,087 B of headroom under the 25 MiB per-file limit; 0 missing references; **0 external
146
+ network dependencies (HERMETIC)**; exit 0.
147
+ - **Env vars:** none (static). The Pages project name / domain is still open (§12).
148
+
149
+ > **The hermetic claim is scoped.** `docs/DEPLOYMENT_DECISION.md` §3 audited `frontend/` (excluding
150
+ > `.tools/`) and found zero occurrences of `fetch(`, `XMLHttpRequest`, `axios`, `EventSource`,
151
+ > `WebSocket`, `/v1/`, `import.meta.env` or `process.env`. `docs/DEPLOYMENT_TOPOLOGY.md` §3.1 narrows
152
+ > this: the "no API calls of any kind" statement holds for every **static** page **except**
153
+ > `mission.html`, which calls the orchestrator. The monorepo `README.md`'s older claim that the
154
+ > frontend is "hermetic — no backend calls" is materially stale.
155
 
156
  ### 3.2 Render — the gateway
157
 
158
+ A deliberately **thin, stateless** orchestrator. It holds **no model, no state, no database**, and
159
+ performs **no auth** (`deploy/render/main.py` module docstring; plan §73/§74). Its responsibilities,
160
+ from `docs/DEPLOYMENT_ARCHITECTURE.md` §2:
161
 
162
  | Responsibility | Detail |
163
  |---|---|
164
  | Schema validation | reject malformed requests before they cost inference |
165
  | Size limits | whole-request body cap, shared with the Codespace |
166
+ | Rate limiting | per-IP count + window — **fairness, NOT a security control** (`docs/DEPLOYMENT_ARCHITECTURE.md` §5.2) |
167
  | CORS allowlist | the Pages origin; **never `*`** |
168
  | Request ids | correlate a request across tiers |
169
+ | Timeouts | sit inside the task budget (§7.5) |
170
  | Secret custody | holds credentials that must never reach the browser |
171
+ | Error translation | upstream failures → the documented error envelope (§7.6) |
172
 
173
  It is **not** a model host. It has **no database, no auth, and no queue**.
174
 
175
+ The gateway's proxied routes (from `deploy/render/main.py`):
176
+
177
+ | Gateway route | Upstream | Notes |
178
+ |---|---|---|
179
+ | `GET /api/health` | answered **locally** | reports the orchestrator's own config; never answers for the Codespace |
180
+ | `POST /api/infer` | `POST {codespace}/v1/analyze` | wake-then-proxy; sets `X-SatQuery-State: waking|ready` |
181
+ | `GET /api/capabilities` | `GET {codespace}/v1/capabilities` | **no second copy** of the capability table |
182
+ | `POST /api/assets` | `POST {codespace}/v1/assets` | raw/multipart body relayed verbatim |
183
+
184
  > **Hard rule.** The gateway must **not** retry `POST /api/infer` on its own — a retry would consume
185
+ > inference a second time. The client decides on retry. The reason is recorded in code
186
+ > (`deploy/render/main.py` docstring; `docs/STEP7_BACKEND_CHAIN_REPORT.md` §11).
187
+
188
+ > **No second capability table.** The gateway proxies `/v1/capabilities` and nothing else decides
189
+ > "what can this deployment do?". The authoritative sources are `core.planner.CAPABILITY_ASSETS` and
190
+ > `SpecialistSpec.requires_assets` (`docs/DEPLOYMENT_ARCHITECTURE.md` §2.2).
191
+
192
+ #### 3.2.1 The CORS allowlist is assembled, not just read
193
+
194
+ `deploy/render/main.py::_allowed_origins` assembles the allowlist in a fixed order:
195
+
196
+ 1. `SATQUERY_ALLOWED_ORIGINS` — the operator's comma-separated list (authoritative for extra origins).
197
+ 2. `_PRODUCTION_ORIGINS` — `https://satquery.pages.dev`, **always present**, so a missing env var
198
+ cannot take the live site down.
199
+ 3. `_DEV_ORIGINS` — 10 explicit `host:port` pairs (`localhost` and `127.0.0.1` × ports
200
+ `3000/5500/5173/8000/8080`), added unless `SATQUERY_ALLOW_DEV_ORIGINS` is one of `0`/`false`/`no`/`""`.
201
+
202
+ A wildcard `*` raises `ValueError` — checked both in `_allowed_origins` and in
203
+ `GatewayConfig.__post_init__`, because `CORSMiddleware` does not run that validator
204
+ (`deploy/render/main.py`). The list is deliberately explicit, never a regex or suffix match, so
205
+ allowing localhost for development cannot admit an arbitrary remote site. The health payload reports
206
+ the **effective** list, so a production deployment can prove from outside that the dev origins were
207
+ turned off.
208
+
209
+ #### 3.2.2 The wake flow
210
+
211
+ `deploy/render/main.py::ensure_codespace_up()` returns `(base_url, woke)`:
212
+
213
+ 1. `GET` the Codespace via the GitHub API (`deploy/render/codespaces.py::get_codespace`).
214
+ 2. If `state != "available"`, `POST .../start` (`start_codespace`; GitHub returns `202`, and `204` is
215
+ also seen in practice).
216
+ 3. Poll `GET {base}/v1/health` until `200` or until `SATQUERY_WAKE_TIMEOUT_S` elapses.
217
+
218
+ Polling knobs: `_WAKE_POLL_INTERVAL_S = 2.0`, `_WAKE_HEALTH_TIMEOUT_S = 10.0`
219
+ (`deploy/render/main.py`). The public base URL is derived by `forwarded_url()`, which prefers the
220
+ Codespace JSON's `web_url` and rewrites its trailing port segment, falling back to
221
+ `https://{name}-{port}.app.github.dev`. That host pattern is an **isolated assumption**: the module's
222
+ own docstring records that it "was **not verifiable from the build environment** (no live Codespace to
223
+ inspect)".
224
 
225
  ### 3.3 GitHub Codespace — the inference tier
226
 
227
  Runs the real inference service: `build_space_app()` from `app/space_app.py`, served by
228
  `deploy/codespace/serve.py` on `$PORT`, in **CPU mode**. It honours the four-endpoint contract
229
  (`/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets`), imports cheaply without torch, reuses
230
+ `app/serving.py` as the composition root, and **degrades rather than crashes** on absent artifacts
231
+ (`docs/DEPLOYMENT_ARCHITECTURE.md` §3.3).
232
+
233
+ The serve entrypoint is deliberately tiny (`deploy/codespace/serve.py`):
234
+
235
+ ```python
236
+ from app.space_app import build_space_app
237
+ import uvicorn
238
+
239
+ app = build_space_app()
240
+
241
+ if __name__ == "__main__":
242
+ port = int(os.environ.get("PORT", "8000"))
243
+ uvicorn.run(app, host="0.0.0.0", port=port)
244
+ ```
245
+
246
+ The composition root (`app/serving.py::build_serving_controller`) resolves `device` from
247
+ `SATQUERY_DEVICE`. It wires three capabilities through the registry's `builders=` seam **without
248
+ editing `configs/base.yaml`**:
249
+
250
+ | Capability | Wired artifact | Why the seam |
251
+ |---|---|---|
252
+ | `change` | `artifacts/change/levir_change_v001/head.pt` | `change.checkpoint_path` is unset; adding it to config would move `Config.hash` |
253
+ | `change_vqa` | `artifacts/change_vqa/run/head.pt` + the same STANet | closes the F2 **train/serve skew** (training and serving must share one detector) |
254
+ | `optical_sar` | CROMA (resolved from the pinned identity) + `artifacts/optical_sar/fusion_head_production_v001/head.pt` | `croma.checkpoint_path` is unset, so the encoder was unreachable by default |
255
+
256
+ The seam is a **call-site argument** (`core/registry.py`'s `builders=` override), not config, so
257
+ `Config.hash` stays `78f1e3700da15aa1` (`app/serving.py` docstring). **Degrade, do not crash:** absent
258
+ artifacts yield `available: false` **with a reason**; *corrupt* artifacts raise `ModelLoadError`. The
259
+ two are deliberately not conflated.
260
+
261
+ #### 3.3.1 The Codespace launcher and its survivability design
262
+
263
+ `.devcontainer/devcontainer.json` sets `postStartCommand: bash deploy/codespace/launch.sh`, so the
264
+ inference server and the tunnel agent start on **every** Codespace start. `launch.sh` is more
265
+ defensive than it looks, and the reasons are recorded in the script:
266
+
267
+ - **Preflight (refuse to start half-configured).** It checks `import yaml, pydantic, fastapi, uvicorn,
268
+ httpx` and `import app.space_app`, exiting non-zero with a diagnostic if either fails. `httpx` is
269
+ checked explicitly because `tunnel_agent.py` imports it directly and it was previously absent from
270
+ `requirements.txt`, so the agent "died instantly and the supervised restart loop hid the error in a
271
+ log file".
272
+ - **Stale-serve detection.** A stamp file (`/tmp/satquery-serve.stamp`) records `rev=<HEAD>
273
+ asset_enabled=<…> asset_dir=<…>`. If the running server's stamp disagrees with the current checkout
274
+ and environment, the serve process is restarted, because "a stale serve process is worse than no
275
+ process: it answers `/v1/health` and `/v1/capabilities` from OLD code".
276
+ - **The tunnel agent is supervised and immortal.** `setsid` alone is not enough in Codespaces — the
277
+ lifecycle shell that runs `postStartCommand` can still reap the process group, which "showed up in
278
+ production as 'the agent announced once, then vanished'". The launcher therefore uses
279
+ `setsid + nohup + </dev/null` around a supervising `while true` wrapper that re-launches the agent
280
+ if it exits, so the agent is "effectively immortal for the life of the Codespace".
281
+ - **Post-launch verification.** After a 4-second wait it checks the agent process is alive and that
282
+ the log contains a successful announce (`announced to hub`), because "backgrounding with all output
283
+ discarded means a crashing agent is completely invisible".
284
+
285
+ > `launch.sh` refers to `bash deploy/codespace/doctor.sh` in two diagnostics. `doctor.sh` is **not
286
+ > present in the monorepo working copy**; it lives in the deployed `SatQuery-Inference` checkout.
287
+ > `UNKNOWN — not established from the available evidence` whether it is present in that repository, as
288
+ > the private repository was not readable for this documentation pass.
289
+
290
+ #### 3.3.2 The `warm_cache.py` pre-warm
291
+
292
+ `deploy/codespace/post_create.sh` (`postCreateCommand`) installs the lean CPU requirements and runs
293
+ `python deploy/codespace/warm_cache.py`, which pre-downloads the pinned HF models into the HF cache so
294
+ the first `/v1/analyze` is fast. It is idempotent, reports per-model OK/SKIPPED/FAILED status, and
295
+ never aborts on a single miss (`deploy/codespace/README.md`).
296
 
297
  ### 3.4 Hugging Face — the model tier
298
 
299
  Holds the six trained artifacts and the model card. It is **not** the runtime inference host; the
300
+ Codespace resolves the pinned backbones from the Hub at run time. At reconnaissance the public
301
+ repository `thundercode/SatQuery` contained **two files only** (`.gitattributes` and a 25-byte
302
+ `README.md`) — the model card / weights publication is a separate workstream from this deployment.
303
+
304
+ ---
305
 
306
  ## 4. Why the transport is an outbound tunnel
307
 
308
  The inference host is a Codespace in a **private** repository. A forwarded port for a private repo
309
  returns **`302`**, so an inbound-forwarding design cannot work. Instead:
310
 
311
+ - the Codespace runs `deploy/codespace/tunnel_agent.py` (from `SatQuery-Inference`);
312
  - the agent **dials out** to `POST /tunnel/agent` and long-polls;
313
  - work is executed against `http://127.0.0.1:8000` **locally**.
314
 
315
+ This inverts the usual direction: the inference host needs **no inbound firewall hole**, and GitHub's
316
+ port-forwarding relay, port visibility and the repository's visibility are all irrelevant. It also
317
+ means the transport is only alive while the agent is polling.
318
 
319
  **Measured:** `GET /api/health` reported `tunnel.agent_connected: true` with a non-zero `completed`
320
  counter, and `POST /api/infer {}` returned `422 invalid_request` with the response header
321
+ `x-satquery-transport: tunnel` (`release/CURRENT_RELEASE_STATE.md` §1; `docs/FINAL_DELIVERY_TODO.md`
322
+ §6 E-03).
323
+
324
+ When the Codespace is stopped, the agent stops polling → `GET /api/health` reports
325
+ `tunnel.agent_connected: false` and `POST /api/infer` parks until `SATQUERY_TUNNEL_TIMEOUT_S` (150 s),
326
+ then returns `tunnel_offline` (503, `recoverable: true`) (`docs/DEPLOYMENT_TOPOLOGY.md` §2).
327
+
328
+ > **The forwarded-port path is dead**, not merely unused: it returns `302` for the private repo. The
329
+ > GitHub-API wake path (`POST /user/codespaces/{name}/start`) still exists in
330
+ > `deploy/render/codespaces.py`, but the tunnel design relies on the agent reconnecting on Codespace
331
+ > start via the devcontainer `postStartCommand`.
332
+
333
+ ---
334
 
335
  ## 5. The full live health payload (VERIFIED, probed)
336
 
 
357
  }
358
  ```
359
 
360
+ Source: `release/CURRENT_RELEASE_STATE.md` §1. The `completed` counter is a live, monotonically
361
+ increasing value — later captures recorded `completed: 314` (`docs/FINAL_DELIVERY_TODO.md` §1.4) and
362
+ `completed: 338` (`docs/FINAL_DELIVERY_REPORT.md` §3). The count is a runtime fact, not a fixed
363
+ constant; do not quote it as a stable figure.
364
+
365
+ Two things in this payload are load-bearing:
366
+
367
+ 1. **`codespace_name` still carries a trailing `\n`.** This is **B-02**, cosmetic and `OPEN`; the wake
368
+ path strips it (`_codespace_name()` calls `.strip()`), so only the `/api/health` reporting payload
369
+ shows the raw value (§8.2).
370
+ 2. **`transport_mode` is `auto`.** This is the root shape of **B-07** (§8.1).
371
+
372
+ ### 5.1 The live capability contract (VERIFIED, probed)
373
+
374
+ `GET /api/capabilities` → `schema_version 1.0`, **six entries, all `available: true`**
375
+ (`release/CURRENT_RELEASE_STATE.md` §1):
376
+
377
+ | task | requires_pair | max_assets | notes |
378
+ |---|---|---|---|
379
+ | `vqa` | false | 1 | SmolVLM weights fetched from the HF Hub on first use |
380
+ | `caption` | false | 1 | SmolVLM weights fetched from the HF Hub on first use |
381
+ | `grounding` | false | 1 | RemoteCLIP encoder fetched from the HF Hub on first use |
382
+ | `change` | true | 2 | — |
383
+ | `change_vqa` | true | 2 | — |
384
+ | `optical_sar` | true | 2 | `modalities: ["optical","sar"]` |
385
+
386
+ The capability table is served by the **single adapter** `app/deployment.py`, derived from the
387
+ registry's spec table plus filesystem presence. The adapter emits only contract vocabulary
388
+ (`loaded`/`absent`/`unavailable`/`not_requested`/`evicted`) and — precisely because it must not load a
389
+ model to answer a metadata request — it **never emits `loaded` or `evicted`**
390
+ (`docs/DEPLOYMENT_ARCHITECTURE.md` §3.3.1). `available: false` always carries a non-null `reason`.
391
+
392
+ ---
393
 
394
  ## 6. Environment variables
395
 
 
401
  | `CODESPACE_PORT` | `8000` | the inference port |
402
  | `SATQUERY_ALLOWED_ORIGINS` | `https://satquery.pages.dev` | CORS allowlist (never `*`) |
403
  | `SATQUERY_DEVICE` | `cpu` | device preference |
404
+ | `SATQUERY_TRANSPORT` | `auto` | tunnel first, then forward (§8.1) |
405
  | `SATQUERY_TUNNEL_TIMEOUT_S` | `150` | how long to wait on the tunnel |
406
  | `SATQUERY_WAKE_TIMEOUT_S` | `120` | how long to wait for a cold start |
407
  | `SATQUERY_UPSTREAM_TIMEOUT_S` | `90` | gateway → upstream budget |
408
+ | `GITHUB_TOKEN` | present | Codespace control (existence only; never recorded here) |
409
  | `PORT` | platform-supplied | Render's own listen port |
410
 
411
+ Source: `docs/DEPLOYMENT_TOPOLOGY.md` (measured 2026-09-25 live note) and
412
+ `release/CURRENT_RELEASE_STATE.md` §1.
413
+
414
+ `render.yaml` in the monorepo declares the blueprint's env vars: `PORT`, `SATQUERY_ALLOWED_ORIGINS`,
415
+ `GITHUB_TOKEN`, `CODESPACE_NAME` (`sync: false` — set in the dashboard), plus `CODESPACE_PORT: "8000"`,
416
+ `SATQUERY_DEVICE: "cpu"`, `SATQUERY_WAKE_TIMEOUT_S: "120"`, `SATQUERY_UPSTREAM_TIMEOUT_S: "90"`
417
+ (`render.yaml`). The blueprint does **not** declare `SATQUERY_TRANSPORT` or `SATQUERY_TUNNEL_TIMEOUT_S`
418
+ — those are set in the live dashboard and are part of the deployed `SatQuery-Backend` revision, not the
419
+ monorepo's stale blueprint.
420
+
421
  > **Measured absence.** There is **no** `SATQUERY_UPSTREAM_URL` and **no** `HF_TOKEN` in the live
422
  > config. The transport is the outbound tunnel, not a forwarded port. This contradicts the older
423
+ > `docs/DEPLOYMENT_TOPOLOGY.md` §3.2 table and `docs/DEPLOYMENT_ARCHITECTURE.md` §4, which predate the
424
+ > tunnel design (`release/CURRENT_RELEASE_STATE.md` §1 note; `docs/FINAL_DELIVERY_TODO.md` §1.7 item 4).
425
 
426
  ### 6.2 Codespace (inference)
427
 
428
  | Variable | Purpose |
429
  |---|---|
430
+ | `PORT` | platform-assigned; **must be read** (historical blocker #2, §9) |
431
  | `SATQUERY_DEVICE` | `cpu` \| `cuda` \| `mps` \| `null`; read **without importing torch** |
432
+ | `SATQUERY_MAX_FILE_BYTES` | per-file cap, shared with Render so the two layers cannot disagree |
433
  | `SATQUERY_ASSET_ENABLED` / `SATQUERY_ASSET_DIR` | both required for `/v1/assets`; **fails closed (503)** otherwise |
434
  | `SATQUERY_ASSET_MAX_FILES` / `SATQUERY_ASSET_TTL_S` | optional handle capacity / lifetime |
435
+ | `SATQUERY_HUB_URL` | the Render orchestrator the tunnel agent dials out to; default `https://satquery-backend-m4yv.onrender.com` |
436
+
437
+ Sources: `docs/DEPLOYMENT_TOPOLOGY.md` §3.3; `deploy/codespace/launch.sh`.
438
+
439
+ `.devcontainer/devcontainer.json` sets `containerEnv`: `SATQUERY_DEVICE=cpu`, `PORT=8000`,
440
+ `SATQUERY_ASSET_ENABLED=1`, `SATQUERY_ASSET_DIR=/tmp/satquery-assets`. The launcher re-exports the
441
+ asset variables on every start because `containerEnv` is only applied when the container is
442
+ **created** — "setting it there alone would leave an already-running Codespace unconfigured until a
443
+ rebuild. This script runs on every start and is therefore the effective source of truth"
444
+ (`deploy/codespace/launch.sh`).
445
+
446
+ Asset-store defaults, from `docs/DEPLOYMENT_ARCHITECTURE.md` §4: handle capacity `32`, TTL `900 s`. A
447
+ malformed or non-positive value falls back to the default rather than becoming a zero TTL. The store
448
+ **refuses rather than evicts** a live handle, so a full store answers `503` (ambiguous with an
449
+ unconfigured store — see `docs/DEPLOYMENT_ARCHITECTURE.md` §5.1).
450
 
451
  ### 6.3 Config-loader environment overrides
452
 
453
+ Two registry values can be overridden from the environment **without editing the YAML**
454
  (`core/config.py`):
455
 
456
  | Variable | Effect |
 
458
  | `SATQUERY_PRECISION` | overrides `training.precision` |
459
  | `SATQUERY_TORCH_COMPILE` | overrides `deployment.torch_compile` (`"true"` → `True`) |
460
 
461
+ Both are still validated by the loader. Setting `SATQUERY_TORCH_COMPILE=true` **fails startup**,
462
+ because finding **C-8** forbids `torch.compile` on the (historical) ZeroGPU target — and the loader
463
+ hard-fails on `deployment.torch_compile is True` (`core/config.py`; `configs/deploy.yaml` header;
464
+ `docs/STEP8_FINAL_CONFORMANCE_AUDIT.md` §6.1). This is an example of the loader refusing an incoherent
465
+ configuration rather than silently accepting it.
466
+
467
+ ### 6.4 The environment-variable vocabulary, and where it moved
468
 
469
+ The active design kept the **env-var vocabulary** and moved only the host names. The superseded
470
+ design used `SATQUERY_SPACE_URL`; the tunnel design uses the Codespace name/port pair plus
471
+ `SATQUERY_HUB_URL` on the inference side (`docs/DEPLOYMENT_TOPOLOGY.md` §5). The older
472
+ `SATQUERY_UPSTREAM_URL` name is **not** set live.
473
+
474
+ ---
475
+
476
+ ## 7. Deploy mechanics per tier
477
 
478
  | Tier | Mechanism |
479
  |---|---|
 
482
  | Inference → Codespace | `deploy/codespace/serve.py` binds `build_space_app()` to `$PORT`; `.devcontainer/` forwards 8000 and starts the tunnel agent via `postStartCommand` |
483
  | Repository writes | the **GitHub Git Data API** — blob → tree → commit → `PATCH` ref |
484
 
485
+ ### 7.1 Frontend deploy (measured)
486
+
487
+ ```bash
488
+ cd C:/Users/anish/satquery-ai
489
+
490
+ node scripts/stage_pages.mjs \
491
+ --out=.deploy/dist-final \
492
+ --include=_headers \
493
+ --include=robots.txt \
494
+ --include=assets/img/eo/provenance.json \
495
+ --include=assets/img/eo/CREDITS.md
496
+
497
+ npx wrangler pages deploy "C:/Users/anish/satquery-ai/.deploy/dist-final" --project-name <name>
498
+ ```
499
+
500
+ (`docs/DEPLOYMENT_DECISION.md` §7.) `_headers` and `robots.txt` must be **force-included** because no
501
+ page references them; `provenance.json` and `CREDITS.md` likewise. The measured staging result is
502
+ quoted in §3.1.
503
+
504
+ > **`_headers` cannot un-cache an asset — it concatenates.** See §10 for the Cloudflare trap and the
505
+ > cache-busting consequence (the EO pair was renamed to new `-720` URLs rather than given a new rule).
506
+
507
+ ### 7.2 Backend deploy
508
+
509
+ `render.yaml` is the blueprint: `runtime: python`, `plan: free`, `buildCommand: pip install -r
510
+ deploy/render/requirements.txt`, `startCommand: uvicorn deploy.render.main:app --host 0.0.0.0 --port
511
+ $PORT`, `healthCheckPath: /api/health`. The deployed `SatQuery-Backend` repository is the source of
512
+ truth; the monorepo's `render.yaml` is a snapshot of the tunnel-less revision.
513
+
514
+ ### 7.3 Inference deploy
515
+
516
+ `deploy/codespace/serve.py` binds `build_space_app()` to `$PORT`. `.devcontainer/devcontainer.json`
517
+ forwards `8000` as **public** and runs `launch.sh` on every start. The launcher's stale-serve detection
518
+ (§3.3.1) means a code or environment change causes the running server to be restarted rather than left
519
+ answering from old code.
520
+
521
+ ### 7.4 Repository writes: the GitHub Git Data API
522
 
523
  Every deployed file is uploaded as a **blob** whose sha256 is **computed locally and verified against
524
+ the uploaded blob**, then assembled into a **tree**, **committed**, and the branch **ref patched**
525
+ (`blob → tree → commit → PATCH ref`). This means:
526
 
527
  - each file is **content-verified** rather than trusted;
528
+ - deletions are expressed explicitly as **`sha: null`** tree entries;
529
  - the deploy is **idempotent** — re-running it with identical content produces no change.
530
 
531
  **Measured:** 9 deployed files were re-read from the API and found **sha256 byte-identical** to the
532
+ local copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`;
533
+ `release/CURRENT_RELEASE_STATE.md` §5).
534
+
535
+ ### 7.5 The timeout relationship (do not invert)
536
+
537
+ ```
538
+ gateway upstream timeout < agent.timeout_seconds ≤ the inference host's own request budget
539
+ ```
540
+
541
+ Both bounds are **derived from the frozen config**, not chosen
542
+ (`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §4.2):
543
+
544
+ | Quantity | Value | Source |
545
+ |---|---|---|
546
+ | `agent.timeout_seconds` | **120 s** | `configs/base.yaml` |
547
+ | `gpu_duration_vqa` | **20 s** | `configs/deploy.yaml` |
548
+ | `gpu_duration_grounding` | **45 s** | `configs/deploy.yaml` |
549
+ | `gpu_duration_change` | **30 s** | `configs/deploy.yaml` |
550
+ | `gpu_duration_optical_sar` | **45 s** | `configs/deploy.yaml` |
551
+ | Largest single `gpu_duration_*` | **45 s** | derived |
552
+
553
+ So the upstream timeout belongs **above 45 s** (the longest a single call may run) and **below 120 s**
554
+ (the host's own request budget). `GatewayConfig.__post_init__` refuses a timeout ≤ 45 s and ≥ 120 s
555
+ (`docs/PHASE19_FINAL_HARDENING.md` §3.2). The live value is `SATQUERY_UPSTREAM_TIMEOUT_S = 90`.
556
+
557
+ ### 7.6 The error contract
558
+
559
+ Upstream failures are wrapped in the v1 envelope
560
+ `{"error": {"code", "message", "detail", "recoverable"}}` (`deploy/render/main.py` docstring;
561
+ `docs/DEPLOYMENT_ARCHITECTURE.md` §2.3):
562
+
563
+ | Condition | Status | `recoverable` | Code |
564
+ |---|---|---|---|
565
+ | Connection error to the Codespace | `502` | `true` | `upstream_unreachable` |
566
+ | Wake times out | `504` | `true` | `wake_timeout` |
567
+ | Non-JSON upstream body | `502` | `true` | `schema_validation_error` |
568
+ | Missing `GITHUB_TOKEN` / `CODESPACE_NAME` | `500` | `false` | `orchestrator_config_error` |
569
+ | Malformed request JSON | `400` | `false` | `invalid_request` |
570
+
571
+ The `code` is passed through **unchanged** — the gateway must not remap the taxonomy in
572
+ `core/errors.py`, because a gateway that remapped codes would make the frontend's error handling
573
+ unpredictable (`docs/DEPLOYMENT_ARCHITECTURE.md` §2.3). A non-JSON upstream error is never relayed
574
+ verbatim (defect **G-4**, `docs/STEP7_BACKEND_CHAIN_REPORT.md` §13).
575
+
576
+ ---
577
 
578
  ## 8. Cold start (documented, not hidden)
579
 
 
587
  | Tunnel wait before falling through | `SATQUERY_TUNNEL_TIMEOUT_S` = 150 s |
588
  | Wake wait | `SATQUERY_WAKE_TIMEOUT_S` = 120 s |
589
  | Upstream budget | `SATQUERY_UPSTREAM_TIMEOUT_S` = 90 s |
590
+ | Codespace idle timeout | 30 min (GitHub REST: `idle_timeout_minutes=30`) |
591
+
592
+ Sources: `docs/DEPLOYMENT_TOPOLOGY.md` §2; `docs/FINAL_DELIVERY_TODO.md` §6 E-04.
593
 
594
  Cold start is **documented rather than papered over**: an honest "this will take a while the first
595
  time" is better than a silent hang.
 
606
  ```
607
 
608
  This is the **root shape** of the observed transient tunnel gap, and it is why a request can appear to
609
+ hang and then fail (`release/CURRENT_RELEASE_STATE.md` §6; `docs/FINAL_DELIVERY_TODO.md` §5 B-07:
610
+ "in `auto` transport mode a tunnel timeout **falls through** to the forward path
611
+ (`SatQuery-Backend/main.py:546`), which then burns `wake_timeout_s=120` on a 302 → the observed 504").
612
+
613
+ A patch (`fix-b07-forward-unavailable.patch`) was authored and verified (`git apply --check` clean,
614
+ `py_compile` clean, applies to the deployed `89d80eaddec5`). It adds:
615
+
616
+ - `forward_unavailable` (**503**, terminal `302`/`401`/`403` on the forward path), and
617
+ - `upstream_timeout` (**504**, tunnel healthy but slow), and
618
+ - the `codespace_name` `.strip()` fix.
619
 
620
  > **Status: B-07 is `OPEN`.** The patch is **prepared but NOT deployed.** The deployed health payload
621
+ > still shows the trailing `\n` and the fallthrough remains live.
622
+
623
+ ### 8.2 B-02 — the trailing newline (`OPEN`, cosmetic)
624
+
625
+ The `/api/health` payload reports `codespace_name` with a trailing `\n`. This is **B-02**, confirmed
626
+ **still live** during the reconnaissance. It is **cosmetic**: the wake path is safe because
627
+ `_codespace_name()` calls `.strip()` (`SatQuery-Backend/main.py:123-124`) and the wake path uses it
628
+ (`main.py:357`); only the health-reporting payload (`main.py:619`) reads the raw env var
629
+ (`docs/FINAL_DELIVERY_TODO.md` §4 P2-T03). Fix = change line 619 to `_codespace_name()`, then Render
630
+ redeploys. **Deferred** because a live-backend redeploy before the demonstration was not judged worth
631
+ the risk. **Status: `OPEN` (cosmetic).**
632
+
633
+ ---
634
 
635
  ## 9. The five historical backend blockers
636
 
637
+ Before any backend could boot, five verified blockers had to be closed. Each was re-verified as a real
638
+ blocker (`docs/DEPLOYMENT_DECISION.md` §8), and the current design closes them:
639
 
640
  | # | Blocker (verified) | How it is closed |
641
  |---|---|---|
642
  | 1 | `requirements.txt` declared no `fastapi` / `uvicorn` / `httpx` / `starlette` | the runtime installs the ASGI stack so `build_space_app()` and the gateway can import |
643
  | 2 | no code read `$PORT` — a platform port would be ignored | `deploy/codespace/serve.py` binds to `$PORT`; Render reads its own |
644
+ | 3 | hand-rolled CORS raised `405` on `OPTIONS`, so browser preflight failed | the gateway registers `OPTIONS` explicitly / uses Starlette's `CORSMiddleware` |
645
  | 4 | module-level `app = create_app()` swallowed config errors into `app = None` | construction errors now **propagate** (fail-fast) instead of leaving a dead `app` |
646
  | 5 | adapter integrity unverified on load (`_adapter_sha256` computed but never compared) | the load path compares the digest against an expected value, or fails startup |
647
 
648
+ `docs/DEPLOYMENT_TOPOLOGY.md` §4 records these honestly as "closed by construction / to be verified on
649
+ the first live run" at the time it was written. The live system subsequently ran and served all six
650
+ tasks (`docs/FINAL_DELIVERY_REPORT.md` §4), which is the evidence that the blockers are closed in
651
+ production.
652
+
653
+ ### 9.1 The defects the first real run found
654
+
655
+ `docs/STEP7_BACKEND_CHAIN_REPORT.md` §13 records that the ASGI layer had never executed, and that
656
+ running it surfaced four live defects immediately:
657
+
658
+ | ID | Defect | Severity | Status |
659
+ |---|---|---|---|
660
+ | **G-1** | `request: Request` never resolved (an in-function import left `Request` out of `__globals__`), so **every POST body was misread as a missing query parameter** and no handler ever ran | **Critical** | **FIXED** |
661
+ | **G-2** | an unsupported `force_task` enum value was forwarded upstream instead of refused locally | High | **FIXED** |
662
+ | **G-3** | an empty `HF_TOKEN` produced `Authorization: Bearer `, which httpx rejects → a crash reported as an upstream failure | High | **FIXED** |
663
+ | **G-4** | a non-JSON upstream error body was relayed verbatim, breaking the error contract and leaking internal text | High | **FIXED** |
664
+
665
+ The lesson recorded there is worth carrying: "the first hour of actually running the gateway found a
666
+ critical defect that had been invisible for as long as nobody could run it". The gateway is the
667
+ validation boundary; a gateway that misreads every body while the tests stay green is exactly the
668
+ failure a documented blocker hides.
669
+
670
+ ---
671
+
672
  ## 10. Platform traps (recorded so they are not rediscovered)
673
 
674
  | Trap | Detail |
675
  |---|---|
676
+ | **Cloudflare `_headers` CONCATENATE** | Two matching rules are **merged, not overridden**. A specific rule nested under a broad `/assets/img/*` rule yields `max-age=604800, …, max-age=0, must-revalidate` — and Chromium takes the **FIRST** `max-age`. The file's own "later rules override" comment is **false**. Measured live 2026-09-25 (`docs/FINAL_DELIVERY_TODO.md` §1.7 item 9). |
677
  | **Cloudflare 308 redirect** | `X.html` → `/X`. Reference the extensionless path. |
678
+ | **Forwarded port returns 302** | for a private repo — this is *why* the tunnel exists (§4). |
679
  | **Tunnel agent must start on boot** | via the devcontainer `postStartCommand`, or a restarted Codespace comes up with `agent_connected: false`. |
680
+ | **Never retry `/api/infer` at the gateway** | a retry consumes inference twice (§3.2). |
681
+ | **`deploy/` is stale and untracked** | not the deployed source (§1.1). |
682
+ | **Edge-cache serves deleted files** | The old EO pair URLs still answer `200` from Cloudflare's edge cache (`CF-Cache-Status: HIT`, `Age: 1076`) although the files are deleted; a cache-busted request returns `404`. Nothing references them (`LIVE_VALIDATION_POSTFIX.md`, "Known residuals"). |
683
+ | **`containerEnv` applies only at container creation** | hence `launch.sh` re-exports the asset variables on every start (§6.2). |
684
+ | **`setsid` alone does not survive `postStartCommand`** | the lifecycle shell can reap the process group; the launcher uses `setsid + nohup + </dev/null` plus a supervising wrapper (§3.3.1). |
685
+
686
+ ---
687
 
688
  ## 11. The superseded design, and what did NOT change
689
 
690
  The earlier design ran inference on an **HF Space with ZeroGPU** (5 GPU-min/day, `@spaces.GPU`
691
+ decoration) behind a **Railway** gateway (`docs/DEPLOYMENT_ARCHITECTURE.md` §1;
692
+ `docs/DEPLOYMENT_TOPOLOGY.md` §5). The active design changes three things:
693
 
694
  1. **CPU-first instead of ZeroGPU.** No code change was required — `device_preference` honours
695
  `SATQUERY_DEVICE` and defaults to CPU, every specialist defaults to `device="cpu"`, and all
696
+ placement is `.to(device)` (never `.cuda()`). ZeroGPU's GPU-minute quota and `@spaces.GPU`
697
+ decoration are no longer on the critical path (`docs/DEPLOYMENT_DECISION.md` §5).
698
  2. **A real, always-buildable inference environment.** A Codespace gives a reproducible container
699
+ without a GPU quota or a Space's cold-start constraint. The wake flow replaces ZeroGPU lazy loading
700
+ as the cold-start story.
701
  3. **No GPU quota to protect at the gateway.** Rate/size limits remain, but as **fairness** controls
702
  rather than quota protection.
703
 
 
707
  |---|---|
708
  | the four-endpoint contract | `/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets` |
709
  | the gateway responsibility table | §3.2 above |
710
+ | the env-var vocabulary | only host names moved (`SATQUERY_SPACE_URL` → the Codespace name/port pair) |
711
  | the config freeze | `78f1e3700da15aa1` |
712
 
713
  `configs/deploy.yaml` still describes the old HF-Space/ZeroGPU target (`platform: huggingface-spaces`,
714
  `sdk: gradio`, `zerogpu: true`, the `gpu_duration_*` values). It is **frozen paperwork**: no Gradio
715
+ runtime exists in code, and editing it would move `Config.hash`. It is left undisturbed
716
+ (`docs/DEPLOYMENT_DECISION.md` §4; `configs/deploy.yaml` header).
717
+
718
+ ### 11.1 The ZeroGPU/Gradio target, in full, and why it is inert
719
+
720
+ `configs/deploy.yaml` carries `registry: false`, which makes its non-membership in the config registry
721
+ machine-readable; `core/config.py` reads exactly one file (`configs/base.yaml`) through a single
722
+ `yaml.safe_load` and never globs `configs/*.yaml`. `scripts/validate_deploy_config.py` asserts the
723
+ manifest is inert and that its `deployment:` block is byte-for-byte equal to `configs/base.yaml`'s.
724
+ There is **no Gradio runtime**: no `import gradio`, no `gr.Blocks`, no `gr.Interface`, and the one
725
+ ZeroGPU code path — `spaces.GPU(duration=…)` inside `decorate_gpu()` — "is never applied to any
726
+ route"; routes use plain `@api.get`/`@api.post` (`docs/DEPLOYMENT_DECISION.md` §4). The real
727
+ entrypoint is FastAPI: `build_space_app()`.
728
+
729
+ > The `spaces` package is not installed, so the `@spaces.GPU(duration=…)` path has **never executed**;
730
+ > `decorate_gpu()` returns an identity decorator when `spaces` is absent, which is the correct CPU
731
+ > behaviour (`docs/PHASE19_FINAL_HARDENING.md` §5.2). **Status: REJECTED (superseded; frozen
732
+ > paperwork only).**
733
+
734
+ ### 11.2 The stale `hf/` docs
735
+
736
+ `hf/SETUP.md` and `hf/README.md` assert that the project "does not own any model weights … ships no
737
+ weights, no binaries, and no large artifacts" and that "this environment has no Hugging Face
738
+ credentials". Both were **false** at release time — six trained artifacts exist
739
+ (`release/CURRENT_RELEASE_STATE.md` §6). This is a documentation defect, not a deployment defect; it is
740
+ recorded in [`LIMITATIONS.md`](LIMITATIONS.md) §6.
741
+
742
+ ### 11.3 The stale monorepo `README.md`
743
+
744
+ The monorepo `README.md` calls the frontend *"hermetic — no backend calls"* (it calls `/api/*` on
745
+ Render), puts Render/Codespace as *"in progress"* (both deployed), describes a 4-endpoint `/v1/*`
746
+ contract (the live gateway contract is `/api/*`), omits the tunnel, and points at the stale untracked
747
+ `deploy/` as the deployment source (`release/CURRENT_RELEASE_STATE.md` §6).
748
+
749
+ ### 11.4 Why the three deploy repos are private
750
+
751
+ `SatQuery-Frontend`, `SatQuery-Backend` and `SatQuery-Inference` are **private**; their links `404` for
752
+ an outside audience. This is **BY DESIGN** — the public umbrella `SatQuery-AI` is the audience-facing
753
+ home. The only public code repository at reconnaissance was `Anish-lab-blip/SatQuery-AI`
754
+ (`docs/FINAL_DELIVERY_TODO.md` §4 P9-T01).
755
+
756
+ ---
757
 
758
  ## 12. Deployment preconditions (open items)
759
 
760
+ 1. **Cloudflare Pages project name / domain** — needed for the deploy command and the `Sitemap:` line
761
+ in `robots.txt`. `sitemap.xml` was deliberately **not** created because it needs a canonical
762
+ production domain (`docs/DEPLOYMENT_DECISION.md` §6).
763
+ 2. **Artifacts present, or capabilities honestly ship `available: false`** — the service degrades
764
  rather than crashes on absent artifacts.
765
+ 3. **`HF_TOKEN` on Render _if_ the HF proxy path is used** — not used in the live config (measured
766
+ absence, §6.1).
767
+ 4. **Codespace `.devcontainer/` forwarding `:8000` and starting the tunnel agent** — both present in
768
+ the live deployment.
769
+ 5. **The five blockers in §9 closed and verified on the first live run** — closed; the live system
770
+ served all six tasks (`docs/FINAL_DELIVERY_REPORT.md` §4).
771
+ 6. **The B-07 patch deployed** — **NOT done**; B-07 remains `OPEN` (§8.1).
772
+
773
+ ---
774
 
775
  ## 13. What is `NOT RUN` / `OPEN` / `BLOCKED` for this topic
776
 
777
  | Item | State |
778
  |---|---|
779
+ | B-07 tunnel fallthrough (`auto` mode ≈ 249 s worst case) | **OPEN** — patch prepared, **NOT deployed** |
780
  | B-02 `codespace_name` trailing `\n` | **OPEN** (cosmetic) |
781
  | A deployed system-level load test | **NOT RUN** |
782
  | Multi-region / HA deployment | **NOT RUN** |
783
  | A production database / persistence | **BY DESIGN** (absent) |
784
+ | Auth / user accounts / queue | **BY DESIGN** (absent; plan §73/§74) |
785
  | The ZeroGPU/Gradio deployment target | **REJECTED** (superseded; frozen paperwork only) |
786
+ | `doctor.sh` present in the deployed Inference repo | `UNKNOWN — not established from the available evidence` |
787
+ | A latency benchmark of the deployed path | **NOT RUN** (see [`PERFORMANCE.md`](PERFORMANCE.md), [`LIMITATIONS.md`](LIMITATIONS.md) §3) |
788
+
789
+ ---
790
 
791
  ## 14. Where the evidence lives
792
 
793
  | Evidence | Location |
794
  |---|---|
795
  | Live health + capabilities probe | `release/CURRENT_RELEASE_STATE.md` §1 |
796
+ | Active topology contract | `docs/DEPLOYMENT_TOPOLOGY.md` |
797
+ | Backend contract (gateway responsibilities, env vocabulary, failure modes) | `docs/DEPLOYMENT_ARCHITECTURE.md` §1.1, §2, §3.3, §4, §5 |
798
+ | Deployment decision + the frozen HF manifest | `docs/DEPLOYMENT_DECISION.md` |
799
+ | Operator runbook (local verification, timeout window, rollback) | `docs/BACKEND_DEPLOYMENT_RUNBOOK.md` |
800
+ | Deployment packaging / open items | `docs/PHASE18_DEPLOYMENT_PACKAGING.md` |
801
+ | Final hardening + ASGI defects | `docs/PHASE19_FINAL_HARDENING.md`, `docs/STEP7_BACKEND_CHAIN_REPORT.md` |
802
  | The undeployed B-07 patch | session scratch: `fix-b07-forward-unavailable.patch` |
803
  | Deployed-artifact integrity check | `verify_deployed_head.py` (session scratch) |
804
+ | Live validation (3 passes, 24 runs) | `.workbuddy-ai/scratch/live_validation/` |
805
+ | Delivery status + blocker register | `docs/FINAL_DELIVERY_TODO.md`, `docs/FINAL_DELIVERY_REPORT.md` |