Image-to-Text
PyTorch
Safetensors
PEFT
English
remote-sensing
satellite-imagery
earth-observation
change-detection
visual-grounding
image-captioning
visual-question-answering
optical-sar-fusion
sar
multimodal
lora
Instructions to use thundercode/SatQuery with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use thundercode/SatQuery with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
release: add docs/DEPLOYMENT.md
Browse files- docs/DEPLOYMENT.md +544 -64
docs/DEPLOYMENT.md
CHANGED
|
@@ -3,18 +3,69 @@
|
|
| 3 |
**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `OPEN` · `DEFERRED` · `BLOCKED` · `BY DESIGN`.
|
| 4 |
|
| 5 |
The live SatQuery AI system runs across **three private repositories** plus one **public umbrella
|
| 6 |
-
repository**
|
| 7 |
-
|
| 8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
> **The single most important trap in this document.** `deploy/` inside the monorepo is **stale and
|
| 11 |
-
> untracked**. It is **not** the deployed source.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
|
| 13 |
---
|
| 14 |
|
| 15 |
-
##
|
| 16 |
|
| 17 |
-
Read from the GitHub API during the release reconnaissance.
|
| 18 |
|
| 19 |
| Component | Repository | Visibility | Branch | Revision | Host |
|
| 20 |
|---|---|---|---|---|---|
|
|
@@ -23,11 +74,32 @@ Read from the GitHub API during the release reconnaissance.
|
|
| 23 |
| Inference | `Anish-lab-blip/SatQuery-Inference` | **private** | `main` | **`5a0936ace491`** | Codespace `potential-space-trout-r4ppw969w45j2pvvw`, port 8000 |
|
| 24 |
| Public umbrella | `Anish-lab-blip/SatQuery-AI` | **public** | `main` | `3dcabd32da41` ("Initial commit") | this release home |
|
| 25 |
| Monorepo (working copy) | `C:/Users/anish/satquery-ai` | local only | `master` | `9d57aed` | **no git remote**; 334 dirty entries |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
-
|
| 28 |
-
the intended home for this release.
|
| 29 |
|
| 30 |
-
##
|
| 31 |
|
| 32 |
```
|
| 33 |
Browser
|
|
@@ -58,62 +130,207 @@ flowchart LR
|
|
| 58 |
R -->|"envelope + error translation"| CF
|
| 59 |
```
|
| 60 |
|
| 61 |
-
|
|
|
|
| 62 |
|
| 63 |
### 3.1 Cloudflare Pages — the static tier
|
| 64 |
|
| 65 |
Serves the frontend. **No backend, no secrets, and no API calls of its own** on the static pages. The
|
| 66 |
-
one exception is the Analyze console, which calls the gateway.
|
| 67 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
|
| 69 |
### 3.2 Render — the gateway
|
| 70 |
|
| 71 |
-
A deliberately **thin, stateless** orchestrator.
|
| 72 |
-
`
|
|
|
|
| 73 |
|
| 74 |
| Responsibility | Detail |
|
| 75 |
|---|---|
|
| 76 |
| Schema validation | reject malformed requests before they cost inference |
|
| 77 |
| Size limits | whole-request body cap, shared with the Codespace |
|
| 78 |
-
| Rate limiting | per-IP count + window — **fairness, NOT a security control** |
|
| 79 |
| CORS allowlist | the Pages origin; **never `*`** |
|
| 80 |
| Request ids | correlate a request across tiers |
|
| 81 |
-
| Timeouts | sit inside the task budget |
|
| 82 |
| Secret custody | holds credentials that must never reach the browser |
|
| 83 |
-
| Error translation | upstream failures → the documented error envelope |
|
| 84 |
|
| 85 |
It is **not** a model host. It has **no database, no auth, and no queue**.
|
| 86 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
> **Hard rule.** The gateway must **not** retry `POST /api/infer` on its own — a retry would consume
|
| 88 |
-
> inference a second time. The client decides on retry.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
### 3.3 GitHub Codespace — the inference tier
|
| 91 |
|
| 92 |
Runs the real inference service: `build_space_app()` from `app/space_app.py`, served by
|
| 93 |
`deploy/codespace/serve.py` on `$PORT`, in **CPU mode**. It honours the four-endpoint contract
|
| 94 |
(`/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets`), imports cheaply without torch, reuses
|
| 95 |
-
`app/serving.py` as the composition root, and **degrades rather than crashes** on absent artifacts
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
|
| 97 |
### 3.4 Hugging Face — the model tier
|
| 98 |
|
| 99 |
Holds the six trained artifacts and the model card. It is **not** the runtime inference host; the
|
| 100 |
-
Codespace resolves the pinned backbones from the Hub at run time.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
|
| 102 |
## 4. Why the transport is an outbound tunnel
|
| 103 |
|
| 104 |
The inference host is a Codespace in a **private** repository. A forwarded port for a private repo
|
| 105 |
returns **`302`**, so an inbound-forwarding design cannot work. Instead:
|
| 106 |
|
| 107 |
-
- the Codespace runs `deploy/codespace/tunnel_agent.py`;
|
| 108 |
- the agent **dials out** to `POST /tunnel/agent` and long-polls;
|
| 109 |
- work is executed against `http://127.0.0.1:8000` **locally**.
|
| 110 |
|
| 111 |
-
This inverts the usual direction: the inference host needs **no inbound firewall hole**
|
| 112 |
-
|
|
|
|
| 113 |
|
| 114 |
**Measured:** `GET /api/health` reported `tunnel.agent_connected: true` with a non-zero `completed`
|
| 115 |
counter, and `POST /api/infer {}` returned `422 invalid_request` with the response header
|
| 116 |
-
`x-satquery-transport: tunnel`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
|
| 118 |
## 5. The full live health payload (VERIFIED, probed)
|
| 119 |
|
|
@@ -140,8 +357,39 @@ counter, and `POST /api/infer {}` returned `422 invalid_request` with the respon
|
|
| 140 |
}
|
| 141 |
```
|
| 142 |
|
| 143 |
-
|
| 144 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
|
| 146 |
## 6. Environment variables
|
| 147 |
|
|
@@ -153,30 +401,56 @@ wake path strips it (`_codespace_name()`), so only the health payload reports th
|
|
| 153 |
| `CODESPACE_PORT` | `8000` | the inference port |
|
| 154 |
| `SATQUERY_ALLOWED_ORIGINS` | `https://satquery.pages.dev` | CORS allowlist (never `*`) |
|
| 155 |
| `SATQUERY_DEVICE` | `cpu` | device preference |
|
| 156 |
-
| `SATQUERY_TRANSPORT` | `auto` | tunnel first, then forward |
|
| 157 |
| `SATQUERY_TUNNEL_TIMEOUT_S` | `150` | how long to wait on the tunnel |
|
| 158 |
| `SATQUERY_WAKE_TIMEOUT_S` | `120` | how long to wait for a cold start |
|
| 159 |
| `SATQUERY_UPSTREAM_TIMEOUT_S` | `90` | gateway → upstream budget |
|
| 160 |
-
| `GITHUB_TOKEN` | present | Codespace control |
|
| 161 |
| `PORT` | platform-supplied | Render's own listen port |
|
| 162 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 163 |
> **Measured absence.** There is **no** `SATQUERY_UPSTREAM_URL` and **no** `HF_TOKEN` in the live
|
| 164 |
> config. The transport is the outbound tunnel, not a forwarded port. This contradicts the older
|
| 165 |
-
> `docs/DEPLOYMENT_TOPOLOGY.md` §3.2 table, which
|
|
|
|
| 166 |
|
| 167 |
### 6.2 Codespace (inference)
|
| 168 |
|
| 169 |
| Variable | Purpose |
|
| 170 |
|---|---|
|
| 171 |
-
| `PORT` | platform-assigned; **must be read** (
|
| 172 |
| `SATQUERY_DEVICE` | `cpu` \| `cuda` \| `mps` \| `null`; read **without importing torch** |
|
| 173 |
-
| `SATQUERY_MAX_FILE_BYTES` | per-file cap, shared with Render |
|
| 174 |
| `SATQUERY_ASSET_ENABLED` / `SATQUERY_ASSET_DIR` | both required for `/v1/assets`; **fails closed (503)** otherwise |
|
| 175 |
| `SATQUERY_ASSET_MAX_FILES` / `SATQUERY_ASSET_TTL_S` | optional handle capacity / lifetime |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
|
| 177 |
### 6.3 Config-loader environment overrides
|
| 178 |
|
| 179 |
-
Two registry values can be overridden from the environment without editing the YAML
|
| 180 |
(`core/config.py`):
|
| 181 |
|
| 182 |
| Variable | Effect |
|
|
@@ -184,10 +458,22 @@ Two registry values can be overridden from the environment without editing the Y
|
|
| 184 |
| `SATQUERY_PRECISION` | overrides `training.precision` |
|
| 185 |
| `SATQUERY_TORCH_COMPILE` | overrides `deployment.torch_compile` (`"true"` → `True`) |
|
| 186 |
|
| 187 |
-
Both are still validated by the loader
|
| 188 |
-
because finding C-8 forbids `torch.compile`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 189 |
|
| 190 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
|
| 192 |
| Tier | Mechanism |
|
| 193 |
|---|---|
|
|
@@ -196,17 +482,98 @@ because finding C-8 forbids `torch.compile`.
|
|
| 196 |
| Inference → Codespace | `deploy/codespace/serve.py` binds `build_space_app()` to `$PORT`; `.devcontainer/` forwards 8000 and starts the tunnel agent via `postStartCommand` |
|
| 197 |
| Repository writes | the **GitHub Git Data API** — blob → tree → commit → `PATCH` ref |
|
| 198 |
|
| 199 |
-
### 7.1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 200 |
|
| 201 |
Every deployed file is uploaded as a **blob** whose sha256 is **computed locally and verified against
|
| 202 |
-
the uploaded blob**, then assembled into a tree, committed, and the branch ref patched
|
|
|
|
| 203 |
|
| 204 |
- each file is **content-verified** rather than trusted;
|
| 205 |
-
- deletions are expressed explicitly as `sha: null` tree entries;
|
| 206 |
- the deploy is **idempotent** — re-running it with identical content produces no change.
|
| 207 |
|
| 208 |
**Measured:** 9 deployed files were re-read from the API and found **sha256 byte-identical** to the
|
| 209 |
-
local copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
|
| 211 |
## 8. Cold start (documented, not hidden)
|
| 212 |
|
|
@@ -220,6 +587,9 @@ shows *"Waking inference engine…"* during this.
|
|
| 220 |
| Tunnel wait before falling through | `SATQUERY_TUNNEL_TIMEOUT_S` = 150 s |
|
| 221 |
| Wake wait | `SATQUERY_WAKE_TIMEOUT_S` = 120 s |
|
| 222 |
| Upstream budget | `SATQUERY_UPSTREAM_TIMEOUT_S` = 90 s |
|
|
|
|
|
|
|
|
|
|
| 223 |
|
| 224 |
Cold start is **documented rather than papered over**: an honest "this will take a while the first
|
| 225 |
time" is better than a silent hang.
|
|
@@ -236,48 +606,98 @@ The forward path to a **private** repo returns `302` quickly — but the wake st
|
|
| 236 |
```
|
| 237 |
|
| 238 |
This is the **root shape** of the observed transient tunnel gap, and it is why a request can appear to
|
| 239 |
-
hang and then fail
|
| 240 |
-
|
| 241 |
-
`
|
| 242 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
|
| 244 |
> **Status: B-07 is `OPEN`.** The patch is **prepared but NOT deployed.** The deployed health payload
|
| 245 |
-
> still shows the trailing `\n`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 246 |
|
| 247 |
## 9. The five historical backend blockers
|
| 248 |
|
| 249 |
-
Before any backend could boot, five verified blockers had to be closed.
|
|
|
|
| 250 |
|
| 251 |
| # | Blocker (verified) | How it is closed |
|
| 252 |
|---|---|---|
|
| 253 |
| 1 | `requirements.txt` declared no `fastapi` / `uvicorn` / `httpx` / `starlette` | the runtime installs the ASGI stack so `build_space_app()` and the gateway can import |
|
| 254 |
| 2 | no code read `$PORT` — a platform port would be ignored | `deploy/codespace/serve.py` binds to `$PORT`; Render reads its own |
|
| 255 |
-
| 3 | hand-rolled CORS raised `405` on `OPTIONS`, so browser preflight failed | the gateway registers `OPTIONS` explicitly / uses Starlette's
|
| 256 |
| 4 | module-level `app = create_app()` swallowed config errors into `app = None` | construction errors now **propagate** (fail-fast) instead of leaving a dead `app` |
|
| 257 |
| 5 | adapter integrity unverified on load (`_adapter_sha256` computed but never compared) | the load path compares the digest against an expected value, or fails startup |
|
| 258 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 259 |
## 10. Platform traps (recorded so they are not rediscovered)
|
| 260 |
|
| 261 |
| Trap | Detail |
|
| 262 |
|---|---|
|
| 263 |
-
| **Cloudflare `_headers` CONCATENATE** | Two matching rules are **merged, not overridden**. A specific rule nested under a broad `/assets/img/*` rule yields `max-age=604800, …, max-age=0, must-revalidate` — and Chromium takes the **FIRST** `max-age`. The file's own "later rules override" comment is **false**. |
|
| 264 |
| **Cloudflare 308 redirect** | `X.html` → `/X`. Reference the extensionless path. |
|
| 265 |
-
| **Forwarded port returns 302** | for a private repo — this is *why* the tunnel exists. |
|
| 266 |
| **Tunnel agent must start on boot** | via the devcontainer `postStartCommand`, or a restarted Codespace comes up with `agent_connected: false`. |
|
| 267 |
-
| **Never retry `/api/infer` at the gateway** | a retry consumes inference twice. |
|
| 268 |
-
| **`deploy/` is stale and untracked** | not the deployed source. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 269 |
|
| 270 |
## 11. The superseded design, and what did NOT change
|
| 271 |
|
| 272 |
The earlier design ran inference on an **HF Space with ZeroGPU** (5 GPU-min/day, `@spaces.GPU`
|
| 273 |
-
decoration) behind a **Railway** gateway.
|
|
|
|
| 274 |
|
| 275 |
1. **CPU-first instead of ZeroGPU.** No code change was required — `device_preference` honours
|
| 276 |
`SATQUERY_DEVICE` and defaults to CPU, every specialist defaults to `device="cpu"`, and all
|
| 277 |
-
placement is `.to(device)` (never `.cuda()`).
|
|
|
|
| 278 |
2. **A real, always-buildable inference environment.** A Codespace gives a reproducible container
|
| 279 |
-
without a GPU quota or a Space's cold-start constraint. The wake flow replaces ZeroGPU lazy
|
| 280 |
-
|
| 281 |
3. **No GPU quota to protect at the gateway.** Rate/size limits remain, but as **fairness** controls
|
| 282 |
rather than quota protection.
|
| 283 |
|
|
@@ -287,39 +707,99 @@ decoration) behind a **Railway** gateway. The active design changes three things
|
|
| 287 |
|---|---|
|
| 288 |
| the four-endpoint contract | `/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets` |
|
| 289 |
| the gateway responsibility table | §3.2 above |
|
| 290 |
-
| the env-var vocabulary | only host names moved (`SATQUERY_SPACE_URL` →
|
| 291 |
| the config freeze | `78f1e3700da15aa1` |
|
| 292 |
|
| 293 |
`configs/deploy.yaml` still describes the old HF-Space/ZeroGPU target (`platform: huggingface-spaces`,
|
| 294 |
`sdk: gradio`, `zerogpu: true`, the `gpu_duration_*` values). It is **frozen paperwork**: no Gradio
|
| 295 |
-
runtime exists in code, and editing it would move `Config.hash`. It is left undisturbed
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 296 |
|
| 297 |
## 12. Deployment preconditions (open items)
|
| 298 |
|
| 299 |
-
1. Cloudflare Pages project name / domain
|
| 300 |
-
|
|
|
|
|
|
|
| 301 |
rather than crashes on absent artifacts.
|
| 302 |
-
3. `HF_TOKEN` on Render
|
| 303 |
-
|
| 304 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 305 |
|
| 306 |
## 13. What is `NOT RUN` / `OPEN` / `BLOCKED` for this topic
|
| 307 |
|
| 308 |
| Item | State |
|
| 309 |
|---|---|
|
| 310 |
-
| B-07 tunnel fallthrough | **OPEN** — patch prepared, NOT deployed |
|
| 311 |
| B-02 `codespace_name` trailing `\n` | **OPEN** (cosmetic) |
|
| 312 |
| A deployed system-level load test | **NOT RUN** |
|
| 313 |
| Multi-region / HA deployment | **NOT RUN** |
|
| 314 |
| A production database / persistence | **BY DESIGN** (absent) |
|
|
|
|
| 315 |
| The ZeroGPU/Gradio deployment target | **REJECTED** (superseded; frozen paperwork only) |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 316 |
|
| 317 |
## 14. Where the evidence lives
|
| 318 |
|
| 319 |
| Evidence | Location |
|
| 320 |
|---|---|
|
| 321 |
| Live health + capabilities probe | `release/CURRENT_RELEASE_STATE.md` §1 |
|
| 322 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 323 |
| The undeployed B-07 patch | session scratch: `fix-b07-forward-unavailable.patch` |
|
| 324 |
| Deployed-artifact integrity check | `verify_deployed_head.py` (session scratch) |
|
| 325 |
-
| Live validation (3 passes) | `.workbuddy-ai/scratch/live_validation/` |
|
|
|
|
|
|
| 3 |
**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `OPEN` · `DEFERRED` · `BLOCKED` · `BY DESIGN`.
|
| 4 |
|
| 5 |
The live SatQuery AI system runs across **three private repositories** plus one **public umbrella
|
| 6 |
+
repository**, serving a **static frontend** on Cloudflare Pages, a **thin orchestrator / gateway** on
|
| 7 |
+
Render, and a **CPU inference service** in a GitHub Codespace reached over an **outbound tunnel**. The
|
| 8 |
+
monorepo working copy is **not** the deployed source.
|
| 9 |
+
|
| 10 |
+
This document is the exhaustive deployment reference: the four tiers, every live revision, every
|
| 11 |
+
environment variable (with measured live values), the deploy mechanics per tier, cold-start
|
| 12 |
+
behaviour, the five historical backend blockers, the platform traps, and the superseded design that
|
| 13 |
+
the active one replaced. It is written to be readable *without* the source tree, but every non-obvious
|
| 14 |
+
claim carries the file it came from.
|
| 15 |
|
| 16 |
> **The single most important trap in this document.** `deploy/` inside the monorepo is **stale and
|
| 17 |
+
> untracked**. It is **not** the deployed source. The deployed backend is `SatQuery-Backend/main.py`;
|
| 18 |
+
> the monorepo's `deploy/render/main.py` is an earlier, tunnel-less revision. Edits must go to the
|
| 19 |
+
> three real repositories, never to the local `deploy/` tree.
|
| 20 |
+
|
| 21 |
+
**Companions.** [`architecture/02-deployment-topology.md`](architecture/02-deployment-topology.md)
|
| 22 |
+
(the long-form topology treatment), [`OPERATIONS.md`](OPERATIONS.md) (running the live system),
|
| 23 |
+
[`SECURITY.md`](SECURITY.md) (the trust boundary),
|
| 24 |
+
[`architecture/10-observability-and-ops.md`](architecture/10-observability-and-ops.md) (health,
|
| 25 |
+
traces and the operator surface), [`LIMITATIONS.md`](LIMITATIONS.md) §4 (operational limitations),
|
| 26 |
+
[`RESEARCH_NOTES.md`](RESEARCH_NOTES.md) §6 (the `auto`-mode fallthrough), and
|
| 27 |
+
[`TESTING.md`](TESTING.md) (how the deploy-time properties are tested).
|
| 28 |
+
|
| 29 |
+
---
|
| 30 |
+
|
| 31 |
+
## 1. How to read this document
|
| 32 |
+
|
| 33 |
+
| Convention | Meaning |
|
| 34 |
+
|---|---|
|
| 35 |
+
| **VERIFIED** | Read from a live endpoint, a git ref, or a file on disk during the release reconnaissance |
|
| 36 |
+
| **MEASURED** | A value with a recorded number and its source |
|
| 37 |
+
| **IMPLEMENTED** | Code exists; whether it ran is stated separately |
|
| 38 |
+
| **NOT RUN** | The work has not been executed |
|
| 39 |
+
| **OPEN** | A known defect or gap that is not closed |
|
| 40 |
+
| **BY DESIGN** | Deliberately absent, and the reason is recorded |
|
| 41 |
+
| **UNKNOWN** | `UNKNOWN — not established from the available evidence` |
|
| 42 |
+
|
| 43 |
+
Every revision, environment variable and finding below was read from a file or a live endpoint. Where
|
| 44 |
+
a value could not be established, the text says so rather than guessing.
|
| 45 |
+
|
| 46 |
+
### 1.1 The three deploy sources versus the working copy
|
| 47 |
+
|
| 48 |
+
| Artifact | Location | Role |
|
| 49 |
+
|---|---|---|
|
| 50 |
+
| Frontend source | `Anish-lab-blip/SatQuery-Frontend` (private) | staged from the monorepo's `frontend/` contents |
|
| 51 |
+
| Backend source | `Anish-lab-blip/SatQuery-Backend` (private) | Render orchestrator (`main.py`, tunnel client) |
|
| 52 |
+
| Inference source | `Anish-lab-blip/SatQuery-Inference` (private) | Codespace FastAPI + `deploy/codespace/tunnel_agent.py` |
|
| 53 |
+
| Public umbrella | `Anish-lab-blip/SatQuery-AI` (public) | the intended release home |
|
| 54 |
+
| Working copy | `C:/Users/anish/satquery-ai` | **local only, no git remote** |
|
| 55 |
+
|
| 56 |
+
The working copy's `deploy/` is untracked (`git ls-files deploy/` returns empty) and stale. Its
|
| 57 |
+
`deploy/render/main.py` is a **tunnel-less** revision (532 lines by the project's own record); the
|
| 58 |
+
deployed `SatQuery-Backend/main.py` is **768 lines** and carries the tunnel client
|
| 59 |
+
(`docs/FINAL_DELIVERY_TODO.md` §1.1). The tunnel agent
|
| 60 |
+
(`deploy/codespace/tunnel_agent.py`) is **not present in the monorepo working copy at all** — it lives
|
| 61 |
+
in the `SatQuery-Inference` repository, and the monorepo's `deploy/codespace/launch.sh` refers to it as
|
| 62 |
+
a path that only exists in the deployed checkout.
|
| 63 |
|
| 64 |
---
|
| 65 |
|
| 66 |
+
## 2. Live revisions (VERIFIED)
|
| 67 |
|
| 68 |
+
Read from the GitHub API during the release reconnaissance (`release/CURRENT_RELEASE_STATE.md` §1).
|
| 69 |
|
| 70 |
| Component | Repository | Visibility | Branch | Revision | Host |
|
| 71 |
|---|---|---|---|---|---|
|
|
|
|
| 74 |
| Inference | `Anish-lab-blip/SatQuery-Inference` | **private** | `main` | **`5a0936ace491`** | Codespace `potential-space-trout-r4ppw969w45j2pvvw`, port 8000 |
|
| 75 |
| Public umbrella | `Anish-lab-blip/SatQuery-AI` | **public** | `main` | `3dcabd32da41` ("Initial commit") | this release home |
|
| 76 |
| Monorepo (working copy) | `C:/Users/anish/satquery-ai` | local only | `master` | `9d57aed` | **no git remote**; 334 dirty entries |
|
| 77 |
+
| Hugging Face | `thundercode/SatQuery` | **public** | `main` | lastModified `2026-09-25T16:26:53Z` | 2 files only: `.gitattributes`, 25-byte `README.md` |
|
| 78 |
+
|
| 79 |
+
Notes that must not be smoothed over:
|
| 80 |
+
|
| 81 |
+
- The public umbrella `SatQuery-AI` contains **only** `README.md` (13 bytes: `# SatQuery-AI`). At the
|
| 82 |
+
time of the reconnaissance it was effectively empty; it is the intended home for this release.
|
| 83 |
+
- The dirty-entry count is a **snapshot**. `release/CURRENT_RELEASE_STATE.md` §1 records **334** dirty
|
| 84 |
+
entries at release reconnaissance; `docs/FINAL_DELIVERY_TODO.md` §1.1 records **323** (294 untracked,
|
| 85 |
+
20 modified, 8 deleted) at an earlier capture. The working tree changed between the two captures.
|
| 86 |
+
- The three deployed repositories are **private**. Their GitHub links return `404` for an outside
|
| 87 |
+
audience. This is **BY DESIGN** (see §11.4).
|
| 88 |
+
- The Hugging Face repository `thundercode/SatQuery` carried **only two files** at reconnaissance
|
| 89 |
+
(`.gitattributes` and a 25-byte `README.md`). It is **not** the runtime inference host; the Codespace
|
| 90 |
+
resolves pinned backbones from the Hub at run time.
|
| 91 |
+
|
| 92 |
+
### 2.1 A note on the "HEAD re-read" verification
|
| 93 |
+
|
| 94 |
+
Nine deployed frontend files were re-read from the GitHub API and found **sha256 byte-identical** to
|
| 95 |
+
the local copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`,
|
| 96 |
+
`release/CURRENT_RELEASE_STATE.md` §5). Three live-validation passes ran against successive HEADs:
|
| 97 |
+
pass 1 against `ff46eba42b18` + `d413d3672311`, passes 2 and 3 against the final HEAD `2d7ae53b482d`.
|
| 98 |
+
No run id is shared between passes.
|
| 99 |
|
| 100 |
+
---
|
|
|
|
| 101 |
|
| 102 |
+
## 3. The four tiers
|
| 103 |
|
| 104 |
```
|
| 105 |
Browser
|
|
|
|
| 130 |
R -->|"envelope + error translation"| CF
|
| 131 |
```
|
| 132 |
|
| 133 |
+
The important inversion: the middle arrow is **outbound from the inference host**, not inbound to it.
|
| 134 |
+
That is the whole reason the design works for a private repository (§4).
|
| 135 |
|
| 136 |
### 3.1 Cloudflare Pages — the static tier
|
| 137 |
|
| 138 |
Serves the frontend. **No backend, no secrets, and no API calls of its own** on the static pages. The
|
| 139 |
+
one exception is the Analyze console (`mission.html`), which calls the gateway.
|
| 140 |
+
|
| 141 |
+
- **Staged by:** `scripts/stage_pages.mjs` (builds a Pages bundle).
|
| 142 |
+
- **Deployed with:** `npx wrangler pages deploy`.
|
| 143 |
+
- **Deploy result (measured, `docs/DEPLOYMENT_DECISION.md` §7):** 60 files staged, 39,173,936 B
|
| 144 |
+
(37.36 MiB) total, largest file `assets/video/satquery-launch-50s.mp4` at 22,710,313 B (21.66 MiB),
|
| 145 |
+
with 3,504,087 B of headroom under the 25 MiB per-file limit; 0 missing references; **0 external
|
| 146 |
+
network dependencies (HERMETIC)**; exit 0.
|
| 147 |
+
- **Env vars:** none (static). The Pages project name / domain is still open (§12).
|
| 148 |
+
|
| 149 |
+
> **The hermetic claim is scoped.** `docs/DEPLOYMENT_DECISION.md` §3 audited `frontend/` (excluding
|
| 150 |
+
> `.tools/`) and found zero occurrences of `fetch(`, `XMLHttpRequest`, `axios`, `EventSource`,
|
| 151 |
+
> `WebSocket`, `/v1/`, `import.meta.env` or `process.env`. `docs/DEPLOYMENT_TOPOLOGY.md` §3.1 narrows
|
| 152 |
+
> this: the "no API calls of any kind" statement holds for every **static** page **except**
|
| 153 |
+
> `mission.html`, which calls the orchestrator. The monorepo `README.md`'s older claim that the
|
| 154 |
+
> frontend is "hermetic — no backend calls" is materially stale.
|
| 155 |
|
| 156 |
### 3.2 Render — the gateway
|
| 157 |
|
| 158 |
+
A deliberately **thin, stateless** orchestrator. It holds **no model, no state, no database**, and
|
| 159 |
+
performs **no auth** (`deploy/render/main.py` module docstring; plan §73/§74). Its responsibilities,
|
| 160 |
+
from `docs/DEPLOYMENT_ARCHITECTURE.md` §2:
|
| 161 |
|
| 162 |
| Responsibility | Detail |
|
| 163 |
|---|---|
|
| 164 |
| Schema validation | reject malformed requests before they cost inference |
|
| 165 |
| Size limits | whole-request body cap, shared with the Codespace |
|
| 166 |
+
| Rate limiting | per-IP count + window — **fairness, NOT a security control** (`docs/DEPLOYMENT_ARCHITECTURE.md` §5.2) |
|
| 167 |
| CORS allowlist | the Pages origin; **never `*`** |
|
| 168 |
| Request ids | correlate a request across tiers |
|
| 169 |
+
| Timeouts | sit inside the task budget (§7.5) |
|
| 170 |
| Secret custody | holds credentials that must never reach the browser |
|
| 171 |
+
| Error translation | upstream failures → the documented error envelope (§7.6) |
|
| 172 |
|
| 173 |
It is **not** a model host. It has **no database, no auth, and no queue**.
|
| 174 |
|
| 175 |
+
The gateway's proxied routes (from `deploy/render/main.py`):
|
| 176 |
+
|
| 177 |
+
| Gateway route | Upstream | Notes |
|
| 178 |
+
|---|---|---|
|
| 179 |
+
| `GET /api/health` | answered **locally** | reports the orchestrator's own config; never answers for the Codespace |
|
| 180 |
+
| `POST /api/infer` | `POST {codespace}/v1/analyze` | wake-then-proxy; sets `X-SatQuery-State: waking|ready` |
|
| 181 |
+
| `GET /api/capabilities` | `GET {codespace}/v1/capabilities` | **no second copy** of the capability table |
|
| 182 |
+
| `POST /api/assets` | `POST {codespace}/v1/assets` | raw/multipart body relayed verbatim |
|
| 183 |
+
|
| 184 |
> **Hard rule.** The gateway must **not** retry `POST /api/infer` on its own — a retry would consume
|
| 185 |
+
> inference a second time. The client decides on retry. The reason is recorded in code
|
| 186 |
+
> (`deploy/render/main.py` docstring; `docs/STEP7_BACKEND_CHAIN_REPORT.md` §11).
|
| 187 |
+
|
| 188 |
+
> **No second capability table.** The gateway proxies `/v1/capabilities` and nothing else decides
|
| 189 |
+
> "what can this deployment do?". The authoritative sources are `core.planner.CAPABILITY_ASSETS` and
|
| 190 |
+
> `SpecialistSpec.requires_assets` (`docs/DEPLOYMENT_ARCHITECTURE.md` §2.2).
|
| 191 |
+
|
| 192 |
+
#### 3.2.1 The CORS allowlist is assembled, not just read
|
| 193 |
+
|
| 194 |
+
`deploy/render/main.py::_allowed_origins` assembles the allowlist in a fixed order:
|
| 195 |
+
|
| 196 |
+
1. `SATQUERY_ALLOWED_ORIGINS` — the operator's comma-separated list (authoritative for extra origins).
|
| 197 |
+
2. `_PRODUCTION_ORIGINS` — `https://satquery.pages.dev`, **always present**, so a missing env var
|
| 198 |
+
cannot take the live site down.
|
| 199 |
+
3. `_DEV_ORIGINS` — 10 explicit `host:port` pairs (`localhost` and `127.0.0.1` × ports
|
| 200 |
+
`3000/5500/5173/8000/8080`), added unless `SATQUERY_ALLOW_DEV_ORIGINS` is one of `0`/`false`/`no`/`""`.
|
| 201 |
+
|
| 202 |
+
A wildcard `*` raises `ValueError` — checked both in `_allowed_origins` and in
|
| 203 |
+
`GatewayConfig.__post_init__`, because `CORSMiddleware` does not run that validator
|
| 204 |
+
(`deploy/render/main.py`). The list is deliberately explicit, never a regex or suffix match, so
|
| 205 |
+
allowing localhost for development cannot admit an arbitrary remote site. The health payload reports
|
| 206 |
+
the **effective** list, so a production deployment can prove from outside that the dev origins were
|
| 207 |
+
turned off.
|
| 208 |
+
|
| 209 |
+
#### 3.2.2 The wake flow
|
| 210 |
+
|
| 211 |
+
`deploy/render/main.py::ensure_codespace_up()` returns `(base_url, woke)`:
|
| 212 |
+
|
| 213 |
+
1. `GET` the Codespace via the GitHub API (`deploy/render/codespaces.py::get_codespace`).
|
| 214 |
+
2. If `state != "available"`, `POST .../start` (`start_codespace`; GitHub returns `202`, and `204` is
|
| 215 |
+
also seen in practice).
|
| 216 |
+
3. Poll `GET {base}/v1/health` until `200` or until `SATQUERY_WAKE_TIMEOUT_S` elapses.
|
| 217 |
+
|
| 218 |
+
Polling knobs: `_WAKE_POLL_INTERVAL_S = 2.0`, `_WAKE_HEALTH_TIMEOUT_S = 10.0`
|
| 219 |
+
(`deploy/render/main.py`). The public base URL is derived by `forwarded_url()`, which prefers the
|
| 220 |
+
Codespace JSON's `web_url` and rewrites its trailing port segment, falling back to
|
| 221 |
+
`https://{name}-{port}.app.github.dev`. That host pattern is an **isolated assumption**: the module's
|
| 222 |
+
own docstring records that it "was **not verifiable from the build environment** (no live Codespace to
|
| 223 |
+
inspect)".
|
| 224 |
|
| 225 |
### 3.3 GitHub Codespace — the inference tier
|
| 226 |
|
| 227 |
Runs the real inference service: `build_space_app()` from `app/space_app.py`, served by
|
| 228 |
`deploy/codespace/serve.py` on `$PORT`, in **CPU mode**. It honours the four-endpoint contract
|
| 229 |
(`/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets`), imports cheaply without torch, reuses
|
| 230 |
+
`app/serving.py` as the composition root, and **degrades rather than crashes** on absent artifacts
|
| 231 |
+
(`docs/DEPLOYMENT_ARCHITECTURE.md` §3.3).
|
| 232 |
+
|
| 233 |
+
The serve entrypoint is deliberately tiny (`deploy/codespace/serve.py`):
|
| 234 |
+
|
| 235 |
+
```python
|
| 236 |
+
from app.space_app import build_space_app
|
| 237 |
+
import uvicorn
|
| 238 |
+
|
| 239 |
+
app = build_space_app()
|
| 240 |
+
|
| 241 |
+
if __name__ == "__main__":
|
| 242 |
+
port = int(os.environ.get("PORT", "8000"))
|
| 243 |
+
uvicorn.run(app, host="0.0.0.0", port=port)
|
| 244 |
+
```
|
| 245 |
+
|
| 246 |
+
The composition root (`app/serving.py::build_serving_controller`) resolves `device` from
|
| 247 |
+
`SATQUERY_DEVICE`. It wires three capabilities through the registry's `builders=` seam **without
|
| 248 |
+
editing `configs/base.yaml`**:
|
| 249 |
+
|
| 250 |
+
| Capability | Wired artifact | Why the seam |
|
| 251 |
+
|---|---|---|
|
| 252 |
+
| `change` | `artifacts/change/levir_change_v001/head.pt` | `change.checkpoint_path` is unset; adding it to config would move `Config.hash` |
|
| 253 |
+
| `change_vqa` | `artifacts/change_vqa/run/head.pt` + the same STANet | closes the F2 **train/serve skew** (training and serving must share one detector) |
|
| 254 |
+
| `optical_sar` | CROMA (resolved from the pinned identity) + `artifacts/optical_sar/fusion_head_production_v001/head.pt` | `croma.checkpoint_path` is unset, so the encoder was unreachable by default |
|
| 255 |
+
|
| 256 |
+
The seam is a **call-site argument** (`core/registry.py`'s `builders=` override), not config, so
|
| 257 |
+
`Config.hash` stays `78f1e3700da15aa1` (`app/serving.py` docstring). **Degrade, do not crash:** absent
|
| 258 |
+
artifacts yield `available: false` **with a reason**; *corrupt* artifacts raise `ModelLoadError`. The
|
| 259 |
+
two are deliberately not conflated.
|
| 260 |
+
|
| 261 |
+
#### 3.3.1 The Codespace launcher and its survivability design
|
| 262 |
+
|
| 263 |
+
`.devcontainer/devcontainer.json` sets `postStartCommand: bash deploy/codespace/launch.sh`, so the
|
| 264 |
+
inference server and the tunnel agent start on **every** Codespace start. `launch.sh` is more
|
| 265 |
+
defensive than it looks, and the reasons are recorded in the script:
|
| 266 |
+
|
| 267 |
+
- **Preflight (refuse to start half-configured).** It checks `import yaml, pydantic, fastapi, uvicorn,
|
| 268 |
+
httpx` and `import app.space_app`, exiting non-zero with a diagnostic if either fails. `httpx` is
|
| 269 |
+
checked explicitly because `tunnel_agent.py` imports it directly and it was previously absent from
|
| 270 |
+
`requirements.txt`, so the agent "died instantly and the supervised restart loop hid the error in a
|
| 271 |
+
log file".
|
| 272 |
+
- **Stale-serve detection.** A stamp file (`/tmp/satquery-serve.stamp`) records `rev=<HEAD>
|
| 273 |
+
asset_enabled=<…> asset_dir=<…>`. If the running server's stamp disagrees with the current checkout
|
| 274 |
+
and environment, the serve process is restarted, because "a stale serve process is worse than no
|
| 275 |
+
process: it answers `/v1/health` and `/v1/capabilities` from OLD code".
|
| 276 |
+
- **The tunnel agent is supervised and immortal.** `setsid` alone is not enough in Codespaces — the
|
| 277 |
+
lifecycle shell that runs `postStartCommand` can still reap the process group, which "showed up in
|
| 278 |
+
production as 'the agent announced once, then vanished'". The launcher therefore uses
|
| 279 |
+
`setsid + nohup + </dev/null` around a supervising `while true` wrapper that re-launches the agent
|
| 280 |
+
if it exits, so the agent is "effectively immortal for the life of the Codespace".
|
| 281 |
+
- **Post-launch verification.** After a 4-second wait it checks the agent process is alive and that
|
| 282 |
+
the log contains a successful announce (`announced to hub`), because "backgrounding with all output
|
| 283 |
+
discarded means a crashing agent is completely invisible".
|
| 284 |
+
|
| 285 |
+
> `launch.sh` refers to `bash deploy/codespace/doctor.sh` in two diagnostics. `doctor.sh` is **not
|
| 286 |
+
> present in the monorepo working copy**; it lives in the deployed `SatQuery-Inference` checkout.
|
| 287 |
+
> `UNKNOWN — not established from the available evidence` whether it is present in that repository, as
|
| 288 |
+
> the private repository was not readable for this documentation pass.
|
| 289 |
+
|
| 290 |
+
#### 3.3.2 The `warm_cache.py` pre-warm
|
| 291 |
+
|
| 292 |
+
`deploy/codespace/post_create.sh` (`postCreateCommand`) installs the lean CPU requirements and runs
|
| 293 |
+
`python deploy/codespace/warm_cache.py`, which pre-downloads the pinned HF models into the HF cache so
|
| 294 |
+
the first `/v1/analyze` is fast. It is idempotent, reports per-model OK/SKIPPED/FAILED status, and
|
| 295 |
+
never aborts on a single miss (`deploy/codespace/README.md`).
|
| 296 |
|
| 297 |
### 3.4 Hugging Face — the model tier
|
| 298 |
|
| 299 |
Holds the six trained artifacts and the model card. It is **not** the runtime inference host; the
|
| 300 |
+
Codespace resolves the pinned backbones from the Hub at run time. At reconnaissance the public
|
| 301 |
+
repository `thundercode/SatQuery` contained **two files only** (`.gitattributes` and a 25-byte
|
| 302 |
+
`README.md`) — the model card / weights publication is a separate workstream from this deployment.
|
| 303 |
+
|
| 304 |
+
---
|
| 305 |
|
| 306 |
## 4. Why the transport is an outbound tunnel
|
| 307 |
|
| 308 |
The inference host is a Codespace in a **private** repository. A forwarded port for a private repo
|
| 309 |
returns **`302`**, so an inbound-forwarding design cannot work. Instead:
|
| 310 |
|
| 311 |
+
- the Codespace runs `deploy/codespace/tunnel_agent.py` (from `SatQuery-Inference`);
|
| 312 |
- the agent **dials out** to `POST /tunnel/agent` and long-polls;
|
| 313 |
- work is executed against `http://127.0.0.1:8000` **locally**.
|
| 314 |
|
| 315 |
+
This inverts the usual direction: the inference host needs **no inbound firewall hole**, and GitHub's
|
| 316 |
+
port-forwarding relay, port visibility and the repository's visibility are all irrelevant. It also
|
| 317 |
+
means the transport is only alive while the agent is polling.
|
| 318 |
|
| 319 |
**Measured:** `GET /api/health` reported `tunnel.agent_connected: true` with a non-zero `completed`
|
| 320 |
counter, and `POST /api/infer {}` returned `422 invalid_request` with the response header
|
| 321 |
+
`x-satquery-transport: tunnel` (`release/CURRENT_RELEASE_STATE.md` §1; `docs/FINAL_DELIVERY_TODO.md`
|
| 322 |
+
§6 E-03).
|
| 323 |
+
|
| 324 |
+
When the Codespace is stopped, the agent stops polling → `GET /api/health` reports
|
| 325 |
+
`tunnel.agent_connected: false` and `POST /api/infer` parks until `SATQUERY_TUNNEL_TIMEOUT_S` (150 s),
|
| 326 |
+
then returns `tunnel_offline` (503, `recoverable: true`) (`docs/DEPLOYMENT_TOPOLOGY.md` §2).
|
| 327 |
+
|
| 328 |
+
> **The forwarded-port path is dead**, not merely unused: it returns `302` for the private repo. The
|
| 329 |
+
> GitHub-API wake path (`POST /user/codespaces/{name}/start`) still exists in
|
| 330 |
+
> `deploy/render/codespaces.py`, but the tunnel design relies on the agent reconnecting on Codespace
|
| 331 |
+
> start via the devcontainer `postStartCommand`.
|
| 332 |
+
|
| 333 |
+
---
|
| 334 |
|
| 335 |
## 5. The full live health payload (VERIFIED, probed)
|
| 336 |
|
|
|
|
| 357 |
}
|
| 358 |
```
|
| 359 |
|
| 360 |
+
Source: `release/CURRENT_RELEASE_STATE.md` §1. The `completed` counter is a live, monotonically
|
| 361 |
+
increasing value — later captures recorded `completed: 314` (`docs/FINAL_DELIVERY_TODO.md` §1.4) and
|
| 362 |
+
`completed: 338` (`docs/FINAL_DELIVERY_REPORT.md` §3). The count is a runtime fact, not a fixed
|
| 363 |
+
constant; do not quote it as a stable figure.
|
| 364 |
+
|
| 365 |
+
Two things in this payload are load-bearing:
|
| 366 |
+
|
| 367 |
+
1. **`codespace_name` still carries a trailing `\n`.** This is **B-02**, cosmetic and `OPEN`; the wake
|
| 368 |
+
path strips it (`_codespace_name()` calls `.strip()`), so only the `/api/health` reporting payload
|
| 369 |
+
shows the raw value (§8.2).
|
| 370 |
+
2. **`transport_mode` is `auto`.** This is the root shape of **B-07** (§8.1).
|
| 371 |
+
|
| 372 |
+
### 5.1 The live capability contract (VERIFIED, probed)
|
| 373 |
+
|
| 374 |
+
`GET /api/capabilities` → `schema_version 1.0`, **six entries, all `available: true`**
|
| 375 |
+
(`release/CURRENT_RELEASE_STATE.md` §1):
|
| 376 |
+
|
| 377 |
+
| task | requires_pair | max_assets | notes |
|
| 378 |
+
|---|---|---|---|
|
| 379 |
+
| `vqa` | false | 1 | SmolVLM weights fetched from the HF Hub on first use |
|
| 380 |
+
| `caption` | false | 1 | SmolVLM weights fetched from the HF Hub on first use |
|
| 381 |
+
| `grounding` | false | 1 | RemoteCLIP encoder fetched from the HF Hub on first use |
|
| 382 |
+
| `change` | true | 2 | — |
|
| 383 |
+
| `change_vqa` | true | 2 | — |
|
| 384 |
+
| `optical_sar` | true | 2 | `modalities: ["optical","sar"]` |
|
| 385 |
+
|
| 386 |
+
The capability table is served by the **single adapter** `app/deployment.py`, derived from the
|
| 387 |
+
registry's spec table plus filesystem presence. The adapter emits only contract vocabulary
|
| 388 |
+
(`loaded`/`absent`/`unavailable`/`not_requested`/`evicted`) and — precisely because it must not load a
|
| 389 |
+
model to answer a metadata request — it **never emits `loaded` or `evicted`**
|
| 390 |
+
(`docs/DEPLOYMENT_ARCHITECTURE.md` §3.3.1). `available: false` always carries a non-null `reason`.
|
| 391 |
+
|
| 392 |
+
---
|
| 393 |
|
| 394 |
## 6. Environment variables
|
| 395 |
|
|
|
|
| 401 |
| `CODESPACE_PORT` | `8000` | the inference port |
|
| 402 |
| `SATQUERY_ALLOWED_ORIGINS` | `https://satquery.pages.dev` | CORS allowlist (never `*`) |
|
| 403 |
| `SATQUERY_DEVICE` | `cpu` | device preference |
|
| 404 |
+
| `SATQUERY_TRANSPORT` | `auto` | tunnel first, then forward (§8.1) |
|
| 405 |
| `SATQUERY_TUNNEL_TIMEOUT_S` | `150` | how long to wait on the tunnel |
|
| 406 |
| `SATQUERY_WAKE_TIMEOUT_S` | `120` | how long to wait for a cold start |
|
| 407 |
| `SATQUERY_UPSTREAM_TIMEOUT_S` | `90` | gateway → upstream budget |
|
| 408 |
+
| `GITHUB_TOKEN` | present | Codespace control (existence only; never recorded here) |
|
| 409 |
| `PORT` | platform-supplied | Render's own listen port |
|
| 410 |
|
| 411 |
+
Source: `docs/DEPLOYMENT_TOPOLOGY.md` (measured 2026-09-25 live note) and
|
| 412 |
+
`release/CURRENT_RELEASE_STATE.md` §1.
|
| 413 |
+
|
| 414 |
+
`render.yaml` in the monorepo declares the blueprint's env vars: `PORT`, `SATQUERY_ALLOWED_ORIGINS`,
|
| 415 |
+
`GITHUB_TOKEN`, `CODESPACE_NAME` (`sync: false` — set in the dashboard), plus `CODESPACE_PORT: "8000"`,
|
| 416 |
+
`SATQUERY_DEVICE: "cpu"`, `SATQUERY_WAKE_TIMEOUT_S: "120"`, `SATQUERY_UPSTREAM_TIMEOUT_S: "90"`
|
| 417 |
+
(`render.yaml`). The blueprint does **not** declare `SATQUERY_TRANSPORT` or `SATQUERY_TUNNEL_TIMEOUT_S`
|
| 418 |
+
— those are set in the live dashboard and are part of the deployed `SatQuery-Backend` revision, not the
|
| 419 |
+
monorepo's stale blueprint.
|
| 420 |
+
|
| 421 |
> **Measured absence.** There is **no** `SATQUERY_UPSTREAM_URL` and **no** `HF_TOKEN` in the live
|
| 422 |
> config. The transport is the outbound tunnel, not a forwarded port. This contradicts the older
|
| 423 |
+
> `docs/DEPLOYMENT_TOPOLOGY.md` §3.2 table and `docs/DEPLOYMENT_ARCHITECTURE.md` §4, which predate the
|
| 424 |
+
> tunnel design (`release/CURRENT_RELEASE_STATE.md` §1 note; `docs/FINAL_DELIVERY_TODO.md` §1.7 item 4).
|
| 425 |
|
| 426 |
### 6.2 Codespace (inference)
|
| 427 |
|
| 428 |
| Variable | Purpose |
|
| 429 |
|---|---|
|
| 430 |
+
| `PORT` | platform-assigned; **must be read** (historical blocker #2, §9) |
|
| 431 |
| `SATQUERY_DEVICE` | `cpu` \| `cuda` \| `mps` \| `null`; read **without importing torch** |
|
| 432 |
+
| `SATQUERY_MAX_FILE_BYTES` | per-file cap, shared with Render so the two layers cannot disagree |
|
| 433 |
| `SATQUERY_ASSET_ENABLED` / `SATQUERY_ASSET_DIR` | both required for `/v1/assets`; **fails closed (503)** otherwise |
|
| 434 |
| `SATQUERY_ASSET_MAX_FILES` / `SATQUERY_ASSET_TTL_S` | optional handle capacity / lifetime |
|
| 435 |
+
| `SATQUERY_HUB_URL` | the Render orchestrator the tunnel agent dials out to; default `https://satquery-backend-m4yv.onrender.com` |
|
| 436 |
+
|
| 437 |
+
Sources: `docs/DEPLOYMENT_TOPOLOGY.md` §3.3; `deploy/codespace/launch.sh`.
|
| 438 |
+
|
| 439 |
+
`.devcontainer/devcontainer.json` sets `containerEnv`: `SATQUERY_DEVICE=cpu`, `PORT=8000`,
|
| 440 |
+
`SATQUERY_ASSET_ENABLED=1`, `SATQUERY_ASSET_DIR=/tmp/satquery-assets`. The launcher re-exports the
|
| 441 |
+
asset variables on every start because `containerEnv` is only applied when the container is
|
| 442 |
+
**created** — "setting it there alone would leave an already-running Codespace unconfigured until a
|
| 443 |
+
rebuild. This script runs on every start and is therefore the effective source of truth"
|
| 444 |
+
(`deploy/codespace/launch.sh`).
|
| 445 |
+
|
| 446 |
+
Asset-store defaults, from `docs/DEPLOYMENT_ARCHITECTURE.md` §4: handle capacity `32`, TTL `900 s`. A
|
| 447 |
+
malformed or non-positive value falls back to the default rather than becoming a zero TTL. The store
|
| 448 |
+
**refuses rather than evicts** a live handle, so a full store answers `503` (ambiguous with an
|
| 449 |
+
unconfigured store — see `docs/DEPLOYMENT_ARCHITECTURE.md` §5.1).
|
| 450 |
|
| 451 |
### 6.3 Config-loader environment overrides
|
| 452 |
|
| 453 |
+
Two registry values can be overridden from the environment **without editing the YAML**
|
| 454 |
(`core/config.py`):
|
| 455 |
|
| 456 |
| Variable | Effect |
|
|
|
|
| 458 |
| `SATQUERY_PRECISION` | overrides `training.precision` |
|
| 459 |
| `SATQUERY_TORCH_COMPILE` | overrides `deployment.torch_compile` (`"true"` → `True`) |
|
| 460 |
|
| 461 |
+
Both are still validated by the loader. Setting `SATQUERY_TORCH_COMPILE=true` **fails startup**,
|
| 462 |
+
because finding **C-8** forbids `torch.compile` on the (historical) ZeroGPU target — and the loader
|
| 463 |
+
hard-fails on `deployment.torch_compile is True` (`core/config.py`; `configs/deploy.yaml` header;
|
| 464 |
+
`docs/STEP8_FINAL_CONFORMANCE_AUDIT.md` §6.1). This is an example of the loader refusing an incoherent
|
| 465 |
+
configuration rather than silently accepting it.
|
| 466 |
+
|
| 467 |
+
### 6.4 The environment-variable vocabulary, and where it moved
|
| 468 |
|
| 469 |
+
The active design kept the **env-var vocabulary** and moved only the host names. The superseded
|
| 470 |
+
design used `SATQUERY_SPACE_URL`; the tunnel design uses the Codespace name/port pair plus
|
| 471 |
+
`SATQUERY_HUB_URL` on the inference side (`docs/DEPLOYMENT_TOPOLOGY.md` §5). The older
|
| 472 |
+
`SATQUERY_UPSTREAM_URL` name is **not** set live.
|
| 473 |
+
|
| 474 |
+
---
|
| 475 |
+
|
| 476 |
+
## 7. Deploy mechanics per tier
|
| 477 |
|
| 478 |
| Tier | Mechanism |
|
| 479 |
|---|---|
|
|
|
|
| 482 |
| Inference → Codespace | `deploy/codespace/serve.py` binds `build_space_app()` to `$PORT`; `.devcontainer/` forwards 8000 and starts the tunnel agent via `postStartCommand` |
|
| 483 |
| Repository writes | the **GitHub Git Data API** — blob → tree → commit → `PATCH` ref |
|
| 484 |
|
| 485 |
+
### 7.1 Frontend deploy (measured)
|
| 486 |
+
|
| 487 |
+
```bash
|
| 488 |
+
cd C:/Users/anish/satquery-ai
|
| 489 |
+
|
| 490 |
+
node scripts/stage_pages.mjs \
|
| 491 |
+
--out=.deploy/dist-final \
|
| 492 |
+
--include=_headers \
|
| 493 |
+
--include=robots.txt \
|
| 494 |
+
--include=assets/img/eo/provenance.json \
|
| 495 |
+
--include=assets/img/eo/CREDITS.md
|
| 496 |
+
|
| 497 |
+
npx wrangler pages deploy "C:/Users/anish/satquery-ai/.deploy/dist-final" --project-name <name>
|
| 498 |
+
```
|
| 499 |
+
|
| 500 |
+
(`docs/DEPLOYMENT_DECISION.md` §7.) `_headers` and `robots.txt` must be **force-included** because no
|
| 501 |
+
page references them; `provenance.json` and `CREDITS.md` likewise. The measured staging result is
|
| 502 |
+
quoted in §3.1.
|
| 503 |
+
|
| 504 |
+
> **`_headers` cannot un-cache an asset — it concatenates.** See §10 for the Cloudflare trap and the
|
| 505 |
+
> cache-busting consequence (the EO pair was renamed to new `-720` URLs rather than given a new rule).
|
| 506 |
+
|
| 507 |
+
### 7.2 Backend deploy
|
| 508 |
+
|
| 509 |
+
`render.yaml` is the blueprint: `runtime: python`, `plan: free`, `buildCommand: pip install -r
|
| 510 |
+
deploy/render/requirements.txt`, `startCommand: uvicorn deploy.render.main:app --host 0.0.0.0 --port
|
| 511 |
+
$PORT`, `healthCheckPath: /api/health`. The deployed `SatQuery-Backend` repository is the source of
|
| 512 |
+
truth; the monorepo's `render.yaml` is a snapshot of the tunnel-less revision.
|
| 513 |
+
|
| 514 |
+
### 7.3 Inference deploy
|
| 515 |
+
|
| 516 |
+
`deploy/codespace/serve.py` binds `build_space_app()` to `$PORT`. `.devcontainer/devcontainer.json`
|
| 517 |
+
forwards `8000` as **public** and runs `launch.sh` on every start. The launcher's stale-serve detection
|
| 518 |
+
(§3.3.1) means a code or environment change causes the running server to be restarted rather than left
|
| 519 |
+
answering from old code.
|
| 520 |
+
|
| 521 |
+
### 7.4 Repository writes: the GitHub Git Data API
|
| 522 |
|
| 523 |
Every deployed file is uploaded as a **blob** whose sha256 is **computed locally and verified against
|
| 524 |
+
the uploaded blob**, then assembled into a **tree**, **committed**, and the branch **ref patched**
|
| 525 |
+
(`blob → tree → commit → PATCH ref`). This means:
|
| 526 |
|
| 527 |
- each file is **content-verified** rather than trusted;
|
| 528 |
+
- deletions are expressed explicitly as **`sha: null`** tree entries;
|
| 529 |
- the deploy is **idempotent** — re-running it with identical content produces no change.
|
| 530 |
|
| 531 |
**Measured:** 9 deployed files were re-read from the API and found **sha256 byte-identical** to the
|
| 532 |
+
local copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`;
|
| 533 |
+
`release/CURRENT_RELEASE_STATE.md` §5).
|
| 534 |
+
|
| 535 |
+
### 7.5 The timeout relationship (do not invert)
|
| 536 |
+
|
| 537 |
+
```
|
| 538 |
+
gateway upstream timeout < agent.timeout_seconds ≤ the inference host's own request budget
|
| 539 |
+
```
|
| 540 |
+
|
| 541 |
+
Both bounds are **derived from the frozen config**, not chosen
|
| 542 |
+
(`docs/BACKEND_DEPLOYMENT_RUNBOOK.md` §4.2):
|
| 543 |
+
|
| 544 |
+
| Quantity | Value | Source |
|
| 545 |
+
|---|---|---|
|
| 546 |
+
| `agent.timeout_seconds` | **120 s** | `configs/base.yaml` |
|
| 547 |
+
| `gpu_duration_vqa` | **20 s** | `configs/deploy.yaml` |
|
| 548 |
+
| `gpu_duration_grounding` | **45 s** | `configs/deploy.yaml` |
|
| 549 |
+
| `gpu_duration_change` | **30 s** | `configs/deploy.yaml` |
|
| 550 |
+
| `gpu_duration_optical_sar` | **45 s** | `configs/deploy.yaml` |
|
| 551 |
+
| Largest single `gpu_duration_*` | **45 s** | derived |
|
| 552 |
+
|
| 553 |
+
So the upstream timeout belongs **above 45 s** (the longest a single call may run) and **below 120 s**
|
| 554 |
+
(the host's own request budget). `GatewayConfig.__post_init__` refuses a timeout ≤ 45 s and ≥ 120 s
|
| 555 |
+
(`docs/PHASE19_FINAL_HARDENING.md` §3.2). The live value is `SATQUERY_UPSTREAM_TIMEOUT_S = 90`.
|
| 556 |
+
|
| 557 |
+
### 7.6 The error contract
|
| 558 |
+
|
| 559 |
+
Upstream failures are wrapped in the v1 envelope
|
| 560 |
+
`{"error": {"code", "message", "detail", "recoverable"}}` (`deploy/render/main.py` docstring;
|
| 561 |
+
`docs/DEPLOYMENT_ARCHITECTURE.md` §2.3):
|
| 562 |
+
|
| 563 |
+
| Condition | Status | `recoverable` | Code |
|
| 564 |
+
|---|---|---|---|
|
| 565 |
+
| Connection error to the Codespace | `502` | `true` | `upstream_unreachable` |
|
| 566 |
+
| Wake times out | `504` | `true` | `wake_timeout` |
|
| 567 |
+
| Non-JSON upstream body | `502` | `true` | `schema_validation_error` |
|
| 568 |
+
| Missing `GITHUB_TOKEN` / `CODESPACE_NAME` | `500` | `false` | `orchestrator_config_error` |
|
| 569 |
+
| Malformed request JSON | `400` | `false` | `invalid_request` |
|
| 570 |
+
|
| 571 |
+
The `code` is passed through **unchanged** — the gateway must not remap the taxonomy in
|
| 572 |
+
`core/errors.py`, because a gateway that remapped codes would make the frontend's error handling
|
| 573 |
+
unpredictable (`docs/DEPLOYMENT_ARCHITECTURE.md` §2.3). A non-JSON upstream error is never relayed
|
| 574 |
+
verbatim (defect **G-4**, `docs/STEP7_BACKEND_CHAIN_REPORT.md` §13).
|
| 575 |
+
|
| 576 |
+
---
|
| 577 |
|
| 578 |
## 8. Cold start (documented, not hidden)
|
| 579 |
|
|
|
|
| 587 |
| Tunnel wait before falling through | `SATQUERY_TUNNEL_TIMEOUT_S` = 150 s |
|
| 588 |
| Wake wait | `SATQUERY_WAKE_TIMEOUT_S` = 120 s |
|
| 589 |
| Upstream budget | `SATQUERY_UPSTREAM_TIMEOUT_S` = 90 s |
|
| 590 |
+
| Codespace idle timeout | 30 min (GitHub REST: `idle_timeout_minutes=30`) |
|
| 591 |
+
|
| 592 |
+
Sources: `docs/DEPLOYMENT_TOPOLOGY.md` §2; `docs/FINAL_DELIVERY_TODO.md` §6 E-04.
|
| 593 |
|
| 594 |
Cold start is **documented rather than papered over**: an honest "this will take a while the first
|
| 595 |
time" is better than a silent hang.
|
|
|
|
| 606 |
```
|
| 607 |
|
| 608 |
This is the **root shape** of the observed transient tunnel gap, and it is why a request can appear to
|
| 609 |
+
hang and then fail (`release/CURRENT_RELEASE_STATE.md` §6; `docs/FINAL_DELIVERY_TODO.md` §5 B-07:
|
| 610 |
+
"in `auto` transport mode a tunnel timeout **falls through** to the forward path
|
| 611 |
+
(`SatQuery-Backend/main.py:546`), which then burns `wake_timeout_s=120` on a 302 → the observed 504").
|
| 612 |
+
|
| 613 |
+
A patch (`fix-b07-forward-unavailable.patch`) was authored and verified (`git apply --check` clean,
|
| 614 |
+
`py_compile` clean, applies to the deployed `89d80eaddec5`). It adds:
|
| 615 |
+
|
| 616 |
+
- `forward_unavailable` (**503**, terminal `302`/`401`/`403` on the forward path), and
|
| 617 |
+
- `upstream_timeout` (**504**, tunnel healthy but slow), and
|
| 618 |
+
- the `codespace_name` `.strip()` fix.
|
| 619 |
|
| 620 |
> **Status: B-07 is `OPEN`.** The patch is **prepared but NOT deployed.** The deployed health payload
|
| 621 |
+
> still shows the trailing `\n` and the fallthrough remains live.
|
| 622 |
+
|
| 623 |
+
### 8.2 B-02 — the trailing newline (`OPEN`, cosmetic)
|
| 624 |
+
|
| 625 |
+
The `/api/health` payload reports `codespace_name` with a trailing `\n`. This is **B-02**, confirmed
|
| 626 |
+
**still live** during the reconnaissance. It is **cosmetic**: the wake path is safe because
|
| 627 |
+
`_codespace_name()` calls `.strip()` (`SatQuery-Backend/main.py:123-124`) and the wake path uses it
|
| 628 |
+
(`main.py:357`); only the health-reporting payload (`main.py:619`) reads the raw env var
|
| 629 |
+
(`docs/FINAL_DELIVERY_TODO.md` §4 P2-T03). Fix = change line 619 to `_codespace_name()`, then Render
|
| 630 |
+
redeploys. **Deferred** because a live-backend redeploy before the demonstration was not judged worth
|
| 631 |
+
the risk. **Status: `OPEN` (cosmetic).**
|
| 632 |
+
|
| 633 |
+
---
|
| 634 |
|
| 635 |
## 9. The five historical backend blockers
|
| 636 |
|
| 637 |
+
Before any backend could boot, five verified blockers had to be closed. Each was re-verified as a real
|
| 638 |
+
blocker (`docs/DEPLOYMENT_DECISION.md` §8), and the current design closes them:
|
| 639 |
|
| 640 |
| # | Blocker (verified) | How it is closed |
|
| 641 |
|---|---|---|
|
| 642 |
| 1 | `requirements.txt` declared no `fastapi` / `uvicorn` / `httpx` / `starlette` | the runtime installs the ASGI stack so `build_space_app()` and the gateway can import |
|
| 643 |
| 2 | no code read `$PORT` — a platform port would be ignored | `deploy/codespace/serve.py` binds to `$PORT`; Render reads its own |
|
| 644 |
+
| 3 | hand-rolled CORS raised `405` on `OPTIONS`, so browser preflight failed | the gateway registers `OPTIONS` explicitly / uses Starlette's `CORSMiddleware` |
|
| 645 |
| 4 | module-level `app = create_app()` swallowed config errors into `app = None` | construction errors now **propagate** (fail-fast) instead of leaving a dead `app` |
|
| 646 |
| 5 | adapter integrity unverified on load (`_adapter_sha256` computed but never compared) | the load path compares the digest against an expected value, or fails startup |
|
| 647 |
|
| 648 |
+
`docs/DEPLOYMENT_TOPOLOGY.md` §4 records these honestly as "closed by construction / to be verified on
|
| 649 |
+
the first live run" at the time it was written. The live system subsequently ran and served all six
|
| 650 |
+
tasks (`docs/FINAL_DELIVERY_REPORT.md` §4), which is the evidence that the blockers are closed in
|
| 651 |
+
production.
|
| 652 |
+
|
| 653 |
+
### 9.1 The defects the first real run found
|
| 654 |
+
|
| 655 |
+
`docs/STEP7_BACKEND_CHAIN_REPORT.md` §13 records that the ASGI layer had never executed, and that
|
| 656 |
+
running it surfaced four live defects immediately:
|
| 657 |
+
|
| 658 |
+
| ID | Defect | Severity | Status |
|
| 659 |
+
|---|---|---|---|
|
| 660 |
+
| **G-1** | `request: Request` never resolved (an in-function import left `Request` out of `__globals__`), so **every POST body was misread as a missing query parameter** and no handler ever ran | **Critical** | **FIXED** |
|
| 661 |
+
| **G-2** | an unsupported `force_task` enum value was forwarded upstream instead of refused locally | High | **FIXED** |
|
| 662 |
+
| **G-3** | an empty `HF_TOKEN` produced `Authorization: Bearer `, which httpx rejects → a crash reported as an upstream failure | High | **FIXED** |
|
| 663 |
+
| **G-4** | a non-JSON upstream error body was relayed verbatim, breaking the error contract and leaking internal text | High | **FIXED** |
|
| 664 |
+
|
| 665 |
+
The lesson recorded there is worth carrying: "the first hour of actually running the gateway found a
|
| 666 |
+
critical defect that had been invisible for as long as nobody could run it". The gateway is the
|
| 667 |
+
validation boundary; a gateway that misreads every body while the tests stay green is exactly the
|
| 668 |
+
failure a documented blocker hides.
|
| 669 |
+
|
| 670 |
+
---
|
| 671 |
+
|
| 672 |
## 10. Platform traps (recorded so they are not rediscovered)
|
| 673 |
|
| 674 |
| Trap | Detail |
|
| 675 |
|---|---|
|
| 676 |
+
| **Cloudflare `_headers` CONCATENATE** | Two matching rules are **merged, not overridden**. A specific rule nested under a broad `/assets/img/*` rule yields `max-age=604800, …, max-age=0, must-revalidate` — and Chromium takes the **FIRST** `max-age`. The file's own "later rules override" comment is **false**. Measured live 2026-09-25 (`docs/FINAL_DELIVERY_TODO.md` §1.7 item 9). |
|
| 677 |
| **Cloudflare 308 redirect** | `X.html` → `/X`. Reference the extensionless path. |
|
| 678 |
+
| **Forwarded port returns 302** | for a private repo — this is *why* the tunnel exists (§4). |
|
| 679 |
| **Tunnel agent must start on boot** | via the devcontainer `postStartCommand`, or a restarted Codespace comes up with `agent_connected: false`. |
|
| 680 |
+
| **Never retry `/api/infer` at the gateway** | a retry consumes inference twice (§3.2). |
|
| 681 |
+
| **`deploy/` is stale and untracked** | not the deployed source (§1.1). |
|
| 682 |
+
| **Edge-cache serves deleted files** | The old EO pair URLs still answer `200` from Cloudflare's edge cache (`CF-Cache-Status: HIT`, `Age: 1076`) although the files are deleted; a cache-busted request returns `404`. Nothing references them (`LIVE_VALIDATION_POSTFIX.md`, "Known residuals"). |
|
| 683 |
+
| **`containerEnv` applies only at container creation** | hence `launch.sh` re-exports the asset variables on every start (§6.2). |
|
| 684 |
+
| **`setsid` alone does not survive `postStartCommand`** | the lifecycle shell can reap the process group; the launcher uses `setsid + nohup + </dev/null` plus a supervising wrapper (§3.3.1). |
|
| 685 |
+
|
| 686 |
+
---
|
| 687 |
|
| 688 |
## 11. The superseded design, and what did NOT change
|
| 689 |
|
| 690 |
The earlier design ran inference on an **HF Space with ZeroGPU** (5 GPU-min/day, `@spaces.GPU`
|
| 691 |
+
decoration) behind a **Railway** gateway (`docs/DEPLOYMENT_ARCHITECTURE.md` §1;
|
| 692 |
+
`docs/DEPLOYMENT_TOPOLOGY.md` §5). The active design changes three things:
|
| 693 |
|
| 694 |
1. **CPU-first instead of ZeroGPU.** No code change was required — `device_preference` honours
|
| 695 |
`SATQUERY_DEVICE` and defaults to CPU, every specialist defaults to `device="cpu"`, and all
|
| 696 |
+
placement is `.to(device)` (never `.cuda()`). ZeroGPU's GPU-minute quota and `@spaces.GPU`
|
| 697 |
+
decoration are no longer on the critical path (`docs/DEPLOYMENT_DECISION.md` §5).
|
| 698 |
2. **A real, always-buildable inference environment.** A Codespace gives a reproducible container
|
| 699 |
+
without a GPU quota or a Space's cold-start constraint. The wake flow replaces ZeroGPU lazy loading
|
| 700 |
+
as the cold-start story.
|
| 701 |
3. **No GPU quota to protect at the gateway.** Rate/size limits remain, but as **fairness** controls
|
| 702 |
rather than quota protection.
|
| 703 |
|
|
|
|
| 707 |
|---|---|
|
| 708 |
| the four-endpoint contract | `/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets` |
|
| 709 |
| the gateway responsibility table | §3.2 above |
|
| 710 |
+
| the env-var vocabulary | only host names moved (`SATQUERY_SPACE_URL` → the Codespace name/port pair) |
|
| 711 |
| the config freeze | `78f1e3700da15aa1` |
|
| 712 |
|
| 713 |
`configs/deploy.yaml` still describes the old HF-Space/ZeroGPU target (`platform: huggingface-spaces`,
|
| 714 |
`sdk: gradio`, `zerogpu: true`, the `gpu_duration_*` values). It is **frozen paperwork**: no Gradio
|
| 715 |
+
runtime exists in code, and editing it would move `Config.hash`. It is left undisturbed
|
| 716 |
+
(`docs/DEPLOYMENT_DECISION.md` §4; `configs/deploy.yaml` header).
|
| 717 |
+
|
| 718 |
+
### 11.1 The ZeroGPU/Gradio target, in full, and why it is inert
|
| 719 |
+
|
| 720 |
+
`configs/deploy.yaml` carries `registry: false`, which makes its non-membership in the config registry
|
| 721 |
+
machine-readable; `core/config.py` reads exactly one file (`configs/base.yaml`) through a single
|
| 722 |
+
`yaml.safe_load` and never globs `configs/*.yaml`. `scripts/validate_deploy_config.py` asserts the
|
| 723 |
+
manifest is inert and that its `deployment:` block is byte-for-byte equal to `configs/base.yaml`'s.
|
| 724 |
+
There is **no Gradio runtime**: no `import gradio`, no `gr.Blocks`, no `gr.Interface`, and the one
|
| 725 |
+
ZeroGPU code path — `spaces.GPU(duration=…)` inside `decorate_gpu()` — "is never applied to any
|
| 726 |
+
route"; routes use plain `@api.get`/`@api.post` (`docs/DEPLOYMENT_DECISION.md` §4). The real
|
| 727 |
+
entrypoint is FastAPI: `build_space_app()`.
|
| 728 |
+
|
| 729 |
+
> The `spaces` package is not installed, so the `@spaces.GPU(duration=…)` path has **never executed**;
|
| 730 |
+
> `decorate_gpu()` returns an identity decorator when `spaces` is absent, which is the correct CPU
|
| 731 |
+
> behaviour (`docs/PHASE19_FINAL_HARDENING.md` §5.2). **Status: REJECTED (superseded; frozen
|
| 732 |
+
> paperwork only).**
|
| 733 |
+
|
| 734 |
+
### 11.2 The stale `hf/` docs
|
| 735 |
+
|
| 736 |
+
`hf/SETUP.md` and `hf/README.md` assert that the project "does not own any model weights … ships no
|
| 737 |
+
weights, no binaries, and no large artifacts" and that "this environment has no Hugging Face
|
| 738 |
+
credentials". Both were **false** at release time — six trained artifacts exist
|
| 739 |
+
(`release/CURRENT_RELEASE_STATE.md` §6). This is a documentation defect, not a deployment defect; it is
|
| 740 |
+
recorded in [`LIMITATIONS.md`](LIMITATIONS.md) §6.
|
| 741 |
+
|
| 742 |
+
### 11.3 The stale monorepo `README.md`
|
| 743 |
+
|
| 744 |
+
The monorepo `README.md` calls the frontend *"hermetic — no backend calls"* (it calls `/api/*` on
|
| 745 |
+
Render), puts Render/Codespace as *"in progress"* (both deployed), describes a 4-endpoint `/v1/*`
|
| 746 |
+
contract (the live gateway contract is `/api/*`), omits the tunnel, and points at the stale untracked
|
| 747 |
+
`deploy/` as the deployment source (`release/CURRENT_RELEASE_STATE.md` §6).
|
| 748 |
+
|
| 749 |
+
### 11.4 Why the three deploy repos are private
|
| 750 |
+
|
| 751 |
+
`SatQuery-Frontend`, `SatQuery-Backend` and `SatQuery-Inference` are **private**; their links `404` for
|
| 752 |
+
an outside audience. This is **BY DESIGN** — the public umbrella `SatQuery-AI` is the audience-facing
|
| 753 |
+
home. The only public code repository at reconnaissance was `Anish-lab-blip/SatQuery-AI`
|
| 754 |
+
(`docs/FINAL_DELIVERY_TODO.md` §4 P9-T01).
|
| 755 |
+
|
| 756 |
+
---
|
| 757 |
|
| 758 |
## 12. Deployment preconditions (open items)
|
| 759 |
|
| 760 |
+
1. **Cloudflare Pages project name / domain** — needed for the deploy command and the `Sitemap:` line
|
| 761 |
+
in `robots.txt`. `sitemap.xml` was deliberately **not** created because it needs a canonical
|
| 762 |
+
production domain (`docs/DEPLOYMENT_DECISION.md` §6).
|
| 763 |
+
2. **Artifacts present, or capabilities honestly ship `available: false`** — the service degrades
|
| 764 |
rather than crashes on absent artifacts.
|
| 765 |
+
3. **`HF_TOKEN` on Render _if_ the HF proxy path is used** — not used in the live config (measured
|
| 766 |
+
absence, §6.1).
|
| 767 |
+
4. **Codespace `.devcontainer/` forwarding `:8000` and starting the tunnel agent** — both present in
|
| 768 |
+
the live deployment.
|
| 769 |
+
5. **The five blockers in §9 closed and verified on the first live run** — closed; the live system
|
| 770 |
+
served all six tasks (`docs/FINAL_DELIVERY_REPORT.md` §4).
|
| 771 |
+
6. **The B-07 patch deployed** — **NOT done**; B-07 remains `OPEN` (§8.1).
|
| 772 |
+
|
| 773 |
+
---
|
| 774 |
|
| 775 |
## 13. What is `NOT RUN` / `OPEN` / `BLOCKED` for this topic
|
| 776 |
|
| 777 |
| Item | State |
|
| 778 |
|---|---|
|
| 779 |
+
| B-07 tunnel fallthrough (`auto` mode ≈ 249 s worst case) | **OPEN** — patch prepared, **NOT deployed** |
|
| 780 |
| B-02 `codespace_name` trailing `\n` | **OPEN** (cosmetic) |
|
| 781 |
| A deployed system-level load test | **NOT RUN** |
|
| 782 |
| Multi-region / HA deployment | **NOT RUN** |
|
| 783 |
| A production database / persistence | **BY DESIGN** (absent) |
|
| 784 |
+
| Auth / user accounts / queue | **BY DESIGN** (absent; plan §73/§74) |
|
| 785 |
| The ZeroGPU/Gradio deployment target | **REJECTED** (superseded; frozen paperwork only) |
|
| 786 |
+
| `doctor.sh` present in the deployed Inference repo | `UNKNOWN — not established from the available evidence` |
|
| 787 |
+
| A latency benchmark of the deployed path | **NOT RUN** (see [`PERFORMANCE.md`](PERFORMANCE.md), [`LIMITATIONS.md`](LIMITATIONS.md) §3) |
|
| 788 |
+
|
| 789 |
+
---
|
| 790 |
|
| 791 |
## 14. Where the evidence lives
|
| 792 |
|
| 793 |
| Evidence | Location |
|
| 794 |
|---|---|
|
| 795 |
| Live health + capabilities probe | `release/CURRENT_RELEASE_STATE.md` §1 |
|
| 796 |
+
| Active topology contract | `docs/DEPLOYMENT_TOPOLOGY.md` |
|
| 797 |
+
| Backend contract (gateway responsibilities, env vocabulary, failure modes) | `docs/DEPLOYMENT_ARCHITECTURE.md` §1.1, §2, §3.3, §4, §5 |
|
| 798 |
+
| Deployment decision + the frozen HF manifest | `docs/DEPLOYMENT_DECISION.md` |
|
| 799 |
+
| Operator runbook (local verification, timeout window, rollback) | `docs/BACKEND_DEPLOYMENT_RUNBOOK.md` |
|
| 800 |
+
| Deployment packaging / open items | `docs/PHASE18_DEPLOYMENT_PACKAGING.md` |
|
| 801 |
+
| Final hardening + ASGI defects | `docs/PHASE19_FINAL_HARDENING.md`, `docs/STEP7_BACKEND_CHAIN_REPORT.md` |
|
| 802 |
| The undeployed B-07 patch | session scratch: `fix-b07-forward-unavailable.patch` |
|
| 803 |
| Deployed-artifact integrity check | `verify_deployed_head.py` (session scratch) |
|
| 804 |
+
| Live validation (3 passes, 24 runs) | `.workbuddy-ai/scratch/live_validation/` |
|
| 805 |
+
| Delivery status + blocker register | `docs/FINAL_DELIVERY_TODO.md`, `docs/FINAL_DELIVERY_REPORT.md` |
|