Image-to-Text
PyTorch
Safetensors
PEFT
English
remote-sensing
satellite-imagery
earth-observation
change-detection
visual-grounding
image-captioning
visual-question-answering
optical-sar-fusion
sar
multimodal
lora
Instructions to use thundercode/SatQuery with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use thundercode/SatQuery with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
release: add README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,493 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SatQuery AI
|
| 2 |
+
|
| 3 |
+
**An interactive vision-language assistant for multimodal remote-sensing image analysis.**
|
| 4 |
+
|
| 5 |
+
Ask a natural-language question about a satellite or aerial image β or a pair of images β and SatQuery
|
| 6 |
+
routes it to the right specialist models, collects evidence, and returns a single confidence-scored
|
| 7 |
+
result envelope. It runs on CPU, is served from a static frontend, and is live at
|
| 8 |
+
**https://satquery.pages.dev**.
|
| 9 |
+
|
| 10 |
+
<p align="center">
|
| 11 |
+
<img src="screenshots/analyze-grounding.png" alt="The SatQuery Analyze console answering a grounding question against live inference" width="820">
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
> **Status: research prototype, pre-1.0.** The architecture is frozen. This repository is the
|
| 15 |
+
> documented public release of the system, its trained artifacts, and its measured results β
|
| 16 |
+
> **including the negative ones.**
|
| 17 |
+
|
| 18 |
---
|
| 19 |
+
|
| 20 |
+
## Table of contents
|
| 21 |
+
|
| 22 |
+
- [Motivation](#motivation)
|
| 23 |
+
- [What the system supports](#what-the-system-supports)
|
| 24 |
+
- [Supported inputs](#supported-inputs)
|
| 25 |
+
- [Architecture](#architecture)
|
| 26 |
+
- [Routing and the execution trace](#routing-and-the-execution-trace)
|
| 27 |
+
- [Real inference vs. the preview path](#real-inference-vs-the-preview-path)
|
| 28 |
+
- [Models](#models)
|
| 29 |
+
- [Measured results](#measured-results)
|
| 30 |
+
- [Live validation](#live-validation)
|
| 31 |
+
- [Installation](#installation)
|
| 32 |
+
- [Local development](#local-development)
|
| 33 |
+
- [Deployment](#deployment)
|
| 34 |
+
- [Reproducibility](#reproducibility)
|
| 35 |
+
- [Known limitations](#known-limitations)
|
| 36 |
+
- [Links](#links)
|
| 37 |
+
- [Citation](#citation)
|
| 38 |
+
|
| 39 |
---
|
| 40 |
+
|
| 41 |
+
## Motivation
|
| 42 |
+
|
| 43 |
+
Remote-sensing analysis is fragmented. Detecting change between two acquisitions, localising an
|
| 44 |
+
object, captioning a scene, answering a question about it, and fusing optical with SAR each live in a
|
| 45 |
+
different model, a different preprocessing convention, and a different output schema. Assembling them
|
| 46 |
+
into one answer means re-solving the same problems β tiling, band handling, coordinate systems,
|
| 47 |
+
confidence β every time.
|
| 48 |
+
|
| 49 |
+
SatQuery AI explores a single hypothesis: **a small deterministic router plus a shared evidence
|
| 50 |
+
contract can make a heterogeneous specialist ensemble behave like one system**, without a large
|
| 51 |
+
language model in the control path. The router *understands* the query. A deterministic policy
|
| 52 |
+
*decides* which specialists run. The specialists *compute*. The evidence engine *proves* the answer.
|
| 53 |
+
|
| 54 |
+
Two design rules follow from that, and they are non-negotiable in the codebase:
|
| 55 |
+
|
| 56 |
+
- **No LLM-generated coordinates. No LLM-generated confidence.** Coordinates come from detection and
|
| 57 |
+
segmentation heads; confidence comes from a calibrated scoring path.
|
| 58 |
+
- **Every result carries an observable execution trace** β never chain-of-thought.
|
| 59 |
+
|
| 60 |
+
## What the system supports
|
| 61 |
+
|
| 62 |
+
Six specialist tasks. All six are reported `available: true` by the live capability contract
|
| 63 |
+
(`GET /api/capabilities`, probed 2026-09-25).
|
| 64 |
+
|
| 65 |
+
| Task | What it answers | Assets |
|
| 66 |
+
|---|---|---|
|
| 67 |
+
| `vqa` | A free-form question about a single scene | 1 |
|
| 68 |
+
| `caption` | A description of a single scene | 1 |
|
| 69 |
+
| `grounding` | *Where* is a described object or region β returns boxes | 1 |
|
| 70 |
+
| `change` | *What changed* between two co-registered acquisitions β returns change regions | 2 |
|
| 71 |
+
| `change_vqa` | A yes/no or short question about a detected change | 2 |
|
| 72 |
+
| `optical_sar` | Joint scene classification from an optical + SAR pair | 2 |
|
| 73 |
+
|
| 74 |
+
## Supported inputs
|
| 75 |
+
|
| 76 |
+
Confirmed by the implementation, not assumed:
|
| 77 |
+
|
| 78 |
+
| Modality | Task(s) | Format |
|
| 79 |
+
|---|---|---|
|
| 80 |
+
| Optical, single image | `vqa`, `caption`, `grounding` | JPEG, PNG, TIFF |
|
| 81 |
+
| Temporal optical pair | `change`, `change_vqa` | Two images of **identical dimensions** |
|
| 82 |
+
| Optical + SAR pair | `optical_sar` | GeoTIFF/TIFF preferred |
|
| 83 |
+
|
| 84 |
+
**Modality is inferred server-side from band count**, not from the file extension: `{1, 2}` bands β
|
| 85 |
+
SAR, `{3, 4, 8, 11, 12, 13}` bands β optical. The browser cannot read band count, so the console warns
|
| 86 |
+
when a submitted pair looks like two ordinary photographs rather than an optical/SAR pair.
|
| 87 |
+
|
| 88 |
+
**Per-file upload limit: 4,194,304 bytes (4 MiB).** Larger files are refused with HTTP 413 β imagery
|
| 89 |
+
must be downscaled first.
|
| 90 |
+
|
| 91 |
+
## Architecture
|
| 92 |
+
|
| 93 |
+
This is the **actually deployed** topology. An older direct-client-to-inference design is superseded.
|
| 94 |
+
|
| 95 |
+
```mermaid
|
| 96 |
+
flowchart TD
|
| 97 |
+
B["Browser<br/>(static console)"] -->|HTTPS| CF["Cloudflare Pages<br/>satquery.pages.dev"]
|
| 98 |
+
CF -->|"HTTPS JSON Β· /api/*"| R["Render<br/>satquery-orchestrator"]
|
| 99 |
+
R -->|"outbound long-poll<br/>POST /tunnel/agent"| T{{"outbound tunnel"}}
|
| 100 |
+
T --> C["GitHub Codespace<br/>FastAPI inference Β· CPU Β· :8000"]
|
| 101 |
+
C --> S["Specialists"]
|
| 102 |
+
S --> M["SmolVLM Β· RemoteCLIP Β· STANet-change<br/>CROMA-fusion Β· MiniLM router"]
|
| 103 |
+
M --> E["Evidence engine<br/>+ temperature scaling"]
|
| 104 |
+
E --> RE["ResultEnvelope"]
|
| 105 |
+
RE -->|"tunnel β Render"| B
|
| 106 |
+
```
|
| 107 |
+
|
| 108 |
+
**Why a tunnel.** The inference host runs in a GitHub Codespace. The forwarded-port path is not
|
| 109 |
+
reachable for a private repo (it returns HTTP 302), so the orchestrator keeps a **long-poll tunnel**:
|
| 110 |
+
the Codespace dials out to `POST /tunnel/agent` and holds the connection; Render queues work onto it.
|
| 111 |
+
`transport_mode` is `auto`, and the tunnel is the live transport.
|
| 112 |
+
|
| 113 |
+
### Repository map
|
| 114 |
+
|
| 115 |
+
| Path | Contents |
|
| 116 |
+
|---|---|
|
| 117 |
+
| `app/` | FastAPI inference service and its composition root |
|
| 118 |
+
| `core/` | Config registry, evidence engine, contracts |
|
| 119 |
+
| `specialists/` | One module per specialist (vqa, caption, grounding, change, optical_sar) |
|
| 120 |
+
| `router/` | MiniLM intent router |
|
| 121 |
+
| `gateway/` | Render orchestration hub (`/api/*`, CORS, wake flow) |
|
| 122 |
+
| `frontend/` | The static console (HTML/CSS/JS) |
|
| 123 |
+
| `configs/base.yaml` | The frozen configuration registry β single source of truth |
|
| 124 |
+
| `evaluation/`, `training/` | Evaluation harnesses and training entry points |
|
| 125 |
+
| `artifacts/` | Trained heads, checkpoints, evaluation outputs, provenance |
|
| 126 |
+
| `docs/` | Architecture, models, benchmarks, deployment, limitations |
|
| 127 |
+
|
| 128 |
+
## Routing and the execution trace
|
| 129 |
+
|
| 130 |
+
Routing is deliberately two-stage, and the split matters:
|
| 131 |
+
|
| 132 |
+
1. **`interpret()` β the reading.** A lexical pass over the query produces a *reading*: task intent,
|
| 133 |
+
modality, temporal requirement, spatial scope, and expected evidence kind. It is
|
| 134 |
+
**asset-count-blind**.
|
| 135 |
+
2. **`chooseTask()` β the dispatch.** The reading is combined with the number of attached assets to
|
| 136 |
+
decide the task actually dispatched. This is why a reading of `change` with **one** asset dispatches
|
| 137 |
+
`change_vqa` β the documented quantifier upgrade.
|
| 138 |
+
|
| 139 |
+
> **A router bug worth recording.** An earlier revision evaluated the temporal rule before the
|
| 140 |
+
> location rule, so *"Where are the built-up areas in this image?"* matched `\bbuilt\b` as a *change*
|
| 141 |
+
> marker and `area` inside *"areas"* as a quantifier. With one asset it collapsed to `vqa` and
|
| 142 |
+
> answered "River". Fixed on 2026-09-25 in `frontend/assets/js/mission.js`; the fix is covered by
|
| 143 |
+
> regression tests and verified live. The same defect existed on a second surface
|
| 144 |
+
> (`SQ.policy` in `core.js`) and was fixed the same day.
|
| 145 |
+
|
| 146 |
+
### The eight execution events
|
| 147 |
+
|
| 148 |
+
The console renders an execution trace built from **eight events**, emitted by the frontend around
|
| 149 |
+
real network calls (`SQ.EVENT_NAMES` in `frontend/assets/js/core.js`):
|
| 150 |
+
|
| 151 |
+
| # | Event | Emitted when |
|
| 152 |
+
|---|---|---|
|
| 153 |
+
| 1 | `QUERY_RECEIVED` | The query and assets are accepted |
|
| 154 |
+
| 2 | `QUERY_UNDERSTOOD` | `interpret()` has produced the reading |
|
| 155 |
+
| 3 | `ROUTE_SELECTED` | `chooseTask()` has selected the dispatched task |
|
| 156 |
+
| 4 | `SPECIALIST_STARTED` | The inference request has been issued |
|
| 157 |
+
| 5 | `SPECIALIST_COMPLETED` | The specialist has returned |
|
| 158 |
+
| 6 | `EVIDENCE_GENERATED` | Evidence items are available |
|
| 159 |
+
| 7 | `CONFIDENCE_COMPUTED` | The calibrated confidence is available |
|
| 160 |
+
| 8 | `RESULT_ASSEMBLED` | The `ResultEnvelope` is complete |
|
| 161 |
+
|
| 162 |
+
These are a **frontend** vocabulary driven by observable events β not a backend protocol and not a
|
| 163 |
+
model's reasoning trace. On live runs the trace bar reaches **94.4444 %** (17/18) and every node is
|
| 164 |
+
marked live; the preview path is the only source of mock-marked nodes.
|
| 165 |
+
|
| 166 |
+
## Real inference vs. the preview path
|
| 167 |
+
|
| 168 |
+
- **Real path (production).** With assets attached, the console calls
|
| 169 |
+
`POST /api/infer` on the Render orchestrator. Every run returns a real `run_*` identifier from the
|
| 170 |
+
inference service. **Live validation recorded 0 mock nodes across 24 live runs.**
|
| 171 |
+
- **Preview path.** With *no* files selected, the console renders a labelled illustrative preview so
|
| 172 |
+
the interface is explorable without the stack awake. Preview nodes are explicitly marked `is-mock`
|
| 173 |
+
and never appear in a live run.
|
| 174 |
+
|
| 175 |
+
The distinction is observable, not asserted: a live run shows `live Β· N evidence Β· transport β¦` and
|
| 176 |
+
zero `.trace__node.is-mock` elements.
|
| 177 |
+
|
| 178 |
+
## Models
|
| 179 |
+
|
| 180 |
+
Six trained artifacts are released. **Four are task heads and two are adapters** β none is a complete
|
| 181 |
+
standalone model, and each documents its backbone dependency. Full detail: [`docs/MODELS.md`](docs/MODELS.md),
|
| 182 |
+
[`MODEL_CARD.md`](MODEL_CARD.md), and the generated [`models/manifest.json`](models/manifest.json).
|
| 183 |
+
|
| 184 |
+
| Task | Backbone (pinned) | Custom component | Artifact | Size | Eval data | Metric | Status |
|
| 185 |
+
|---|---|---|---|---|---|---|---|
|
| 186 |
+
| `change` | STANet-style, ResNet-18 encoder, PAM | change head | `head.pt` | 63,231,009 B | LEVIR-CD-256, test n=2048 | pooled IoU **0.8122** Β· macro IoU **0.8457** Β· pooled F1 **0.8964** | **VERIFIED** |
|
| 187 |
+
| `grounding` | `chendelong/RemoteCLIP` ViT-B/32 @ `bf1d8a3ccf2d` (frozen) | trainable head (2048β512) | `head.pt` | 12,639,041 B | VRSBench, n=16159 | mean_best_IoU **0.2838** Β· recall@0.5 **0.2198** (canonical) | measured β two protocols |
|
| 188 |
+
| `optical_sar` | `antofuller/CROMA` base @ `0dd28e3d633b` | fusion head (2318β512β19) | `head.pt` | 14,427,457 B | BigEarthNet, 19 CLC classes, test n=4000 | accuracy **0.931** Β· macro_F1 **0.434161** | measured β **ruling OPEN** |
|
| 189 |
+
| `change_vqa` | as `change` | change-VQA head | `head.pt` | 5,822,809 B | test n=39686 | accuracy **0.697626** Β· macro_F1 **0.378373** | measured β **ruling OPEN** |
|
| 190 |
+
| `router` | `sentence-transformers/all-MiniLM-L6-v2` @ `1110a243fdf4` | intent adapter | `adapter.pt` | 211,961 B | val n=86 | accuracy **0.965116** | **TEST NOT RUN** |
|
| 191 |
+
| `vqa` / `caption` | `HuggingFaceTB/SmolVLM-500M-Instruct` @ `a7da5b986cb5` | **LoRA** (r=16, Ξ±=32, dropout 0.05) | `adapter_model.safetensors` | 34,798,048 B | frozen 1000-Q subset | exact_match **0.963** Β· F1 **0.96432** | **ACCEPTANCE-REJECTED** |
|
| 192 |
+
|
| 193 |
+
Backbones are third-party and pinned by `repo_id` + `revision` in `configs/base.yaml`; they are
|
| 194 |
+
fetched from the Hugging Face Hub, not redistributed here.
|
| 195 |
+
|
| 196 |
+
## Measured results
|
| 197 |
+
|
| 198 |
+
Every number below traces to an artifact, a test, or a live run. **Nothing here is a system-level
|
| 199 |
+
benchmark β no such benchmark exists** (see [Known limitations](#known-limitations)).
|
| 200 |
+
|
| 201 |
+
| Metric | Value | Split / protocol | Source key | Status |
|
| 202 |
+
|---|---|---|---|---|
|
| 203 |
+
| Change pooled IoU | 0.8122 | LEVIR-CD-256 test, n=2048, thr 0.50 | `metrics.pooled.iou` | **VERIFIED** |
|
| 204 |
+
| Change macro IoU | 0.8457 | same | `metrics.macro.miou` | **VERIFIED** |
|
| 205 |
+
| Change pooled F1 | 0.8964 | same | `metrics.pooled.f1` | **VERIFIED** |
|
| 206 |
+
| Grounding mean_best_IoU (canonical, `head_threshold`) | 0.2838 | VRSBench, n=16159 | `results.head_threshold.mean_best_iou` | measured |
|
| 207 |
+
| Grounding recall@0.5 (canonical, `head_threshold`) | 0.2198 | same | `results.head_threshold.recall.0.50` | measured |
|
| 208 |
+
| Grounding mean_best_IoU (matched6, `head_threshold`) | 0.2566 | VRSBench, n=16159 | `results.head_threshold.mean_best_iou` | measured |
|
| 209 |
+
| Grounding recall@0.5 (matched6, `head_threshold`) | 0.1938 | same | `results.head_threshold.recall.0.50` | measured |
|
| 210 |
+
| Grounding `head_argmax` decode (both protocols) | 0.1215 | same | `results.head_argmax.mean_best_iou` | measured β **worse** |
|
| 211 |
+
| Grounding zero-shot baseline | 0.0972 | same | `results.zero_shot_matched.mean_best_iou` | measured |
|
| 212 |
+
| Optical-SAR accuracy | 0.931 | BigEarthNet, held-out test n=4000 | `accuracy` | measured β ruling **OPEN** |
|
| 213 |
+
| Optical-SAR macro_F1 | 0.434161 | same | `macro_f1` | measured β ruling **OPEN** |
|
| 214 |
+
| Change-VQA accuracy (**test**) | 0.697626 | test n=39686 | `verification.test_accuracy` | measured β ruling **OPEN** |
|
| 215 |
+
| Change-VQA macro_F1 (**test**) | 0.378373 | same | `verification.test_macro_f1` | measured β ruling **OPEN** |
|
| 216 |
+
| Change-VQA accuracy (**test2**) | 0.651469 | second test set | `verification.test2_accuracy` | measured β **lower** |
|
| 217 |
+
| Change-VQA macro_F1 (**test2**) | 0.372309 | second test set | `verification.test2_macro_f1` | measured β **lower** |
|
| 218 |
+
| VLM adapter exact_match | 0.963 | frozen 1000-Q subset | `artifacts/vlm/phase6_closure.json` | USABLE_VERIFIED β **ACCEPTANCE-REJECTED** |
|
| 219 |
+
| VLM adapter F1 | 0.96432 | same | same | USABLE_VERIFIED β **ACCEPTANCE-REJECTED** |
|
| 220 |
+
| Router **overall ungated** accuracy | 0.965116 | val, n=86, corpus-limited | `overall_ungated_accuracy` | **TEST NOT RUN** |
|
| 221 |
+
| System-level end-to-end benchmark | β | β | β | **NOT RUN β none exists** |
|
| 222 |
+
|
| 223 |
+
### Grounding: three decode variants, two protocols
|
| 224 |
+
|
| 225 |
+
The grounding head is evaluated under **two matching protocols** (canonical, matched6) and **three
|
| 226 |
+
decode variants**. Quoting a single number would misrepresent the result, so all of them are listed:
|
| 227 |
+
|
| 228 |
+
| Decode | canonical mean_best_IoU | matched6 mean_best_IoU |
|
| 229 |
+
|---|---|---|
|
| 230 |
+
| `head_threshold` (the headline number) | **0.2838** | **0.2566** |
|
| 231 |
+
| `head_argmax` | 0.1215 | 0.1215 |
|
| 232 |
+
| `zero_shot_matched` (baseline, no head) | 0.0972 | 0.0972 |
|
| 233 |
+
|
| 234 |
+
The head clears the zero-shot baseline, but only the threshold decode is meaningfully above it β the
|
| 235 |
+
argmax decode (0.1215) is barely better than zero-shot. The absolute level is modest either way:
|
| 236 |
+
**grounding is useful, not solved.**
|
| 237 |
+
|
| 238 |
+
### Fusion: measured but the ruling is open
|
| 239 |
+
|
| 240 |
+
Optical-SAR fusion reaches **0.931 accuracy** on a 19-class held-out set of 4,000 β but only
|
| 241 |
+
**0.434 macro_F1**. Those two numbers describe very different things: the model is accurate on
|
| 242 |
+
frequent classes and weak on rare ones. The acceptance ruling for this head is **OPEN**, and the
|
| 243 |
+
headline accuracy must never be quoted without the macro_F1 beside it.
|
| 244 |
+
|
| 245 |
+
### Change-VQA: two test sets, and they disagree
|
| 246 |
+
|
| 247 |
+
`artifacts/change_vqa/run/PROMOTION.json` records **two** test evaluations:
|
| 248 |
+
|
| 249 |
+
| Split | accuracy | macro_F1 |
|
| 250 |
+
|---|---|---|
|
| 251 |
+
| `test` | 0.697626 | 0.378373 |
|
| 252 |
+
| `test2` | **0.651469** | 0.372309 |
|
| 253 |
+
|
| 254 |
+
The `test` numbers are the higher pair. Both are reported here; quoting only `test` would overstate
|
| 255 |
+
the result. The acceptance ruling is **OPEN**.
|
| 256 |
+
|
| 257 |
+
### Phase 6 / VLM: deployment success β model acceptance
|
| 258 |
+
|
| 259 |
+
The Phase-6 SmolVLM LoRA adapter reaches **exact_match 0.963** and **F1 0.96432** on a frozen
|
| 260 |
+
1,000-question subset. It is marked **USABLE_VERIFIED** and **ACCEPTANCE-REJECTED**.
|
| 261 |
+
|
| 262 |
+
Those two verdicts are not in conflict, and the distinction is the point:
|
| 263 |
+
|
| 264 |
+
- **USABLE_VERIFIED** β the adapter loads, runs, and produces the measured numbers in the deployed
|
| 265 |
+
pipeline.
|
| 266 |
+
- **ACCEPTANCE-REJECTED** β the change did not clear the project's own pre-registered acceptance bar.
|
| 267 |
+
|
| 268 |
+
A model can be a working engineering artifact and a rejected research result at the same time.
|
| 269 |
+
This release keeps both labels.
|
| 270 |
+
|
| 271 |
+
### Calibration: it got worse, and we say so
|
| 272 |
+
|
| 273 |
+
The `change_vqa` confidence path applies temperature scaling (`T = 0.9773`). Measured on the
|
| 274 |
+
validation split (n=16441):
|
| 275 |
+
|
| 276 |
+
| | ECE |
|
| 277 |
+
|---|---|
|
| 278 |
+
| Before temperature scaling | **0.013755** |
|
| 279 |
+
| After temperature scaling | **0.014929** |
|
| 280 |
+
|
| 281 |
+
**Calibration did not improve β it moved slightly worse.** The scaling is retained because it is part
|
| 282 |
+
of the frozen configuration, not because it helped. The reliability curve plotted on the Benchmark page
|
| 283 |
+
is explicitly labelled as the **pre-scaling** diagram so a reader cannot mistake it for the calibrated
|
| 284 |
+
result.
|
| 285 |
+
|
| 286 |
+
## Live validation
|
| 287 |
+
|
| 288 |
+
Validation drove the **production site** in a headed browser, one upload per case, with per-case
|
| 289 |
+
screenshots and recorded run identifiers.
|
| 290 |
+
|
| 291 |
+
| Property | Result |
|
| 292 |
+
|---|---|
|
| 293 |
+
| Independent full passes | **3** |
|
| 294 |
+
| Cases per pass | 8 (6 regression + 2 defect) |
|
| 295 |
+
| Passes at 8/8 | **3 of 3** |
|
| 296 |
+
| Live runs executed | **24** |
|
| 297 |
+
| Correct dispatches | **24** |
|
| 298 |
+
| Mock-node contamination | **0** on every live run |
|
| 299 |
+
| Trace fill | 94.4444 % on every live run |
|
| 300 |
+
| Frontend regression suite | **106 passed** (`tests/unit/test_frontend_live_wiring.py`) |
|
| 301 |
+
|
| 302 |
+
Each pass produced **fresh run identifiers** β no run id is shared between passes.
|
| 303 |
+
|
| 304 |
+
The harness asserts the form state **before** dispatch β that the query box really holds the intended
|
| 305 |
+
query, that `#obsTail` reads `ready`, and that both frames are attached for pair tasks. This matters:
|
| 306 |
+
an earlier harness revision typed with synthetic key events that Chrome silently drops when the window
|
| 307 |
+
lacks OS focus, so it dispatched the page's *default* query and still recorded a "result". The
|
| 308 |
+
assertions exist because of that failure.
|
| 309 |
+
|
| 310 |
+
### Representative real run IDs
|
| 311 |
+
|
| 312 |
+
One full pass (pass 3 of 3) β the same pass the screenshots below are drawn from. Run identifiers
|
| 313 |
+
are fresh on every pass; the other two passes recorded different ids.
|
| 314 |
+
|
| 315 |
+
| Case | Query | Dispatched | Run ID |
|
| 316 |
+
|---|---|---|---|
|
| 317 |
+
| vqa | What type of terrain dominates this scene? | `vqa` | `run_fef26e91e7e6` |
|
| 318 |
+
| caption | Describe the main visual characteristics of this scene. | `caption` | `run_96281bdfcc08` |
|
| 319 |
+
| grounding | Where are the visible buildings in this image? | `grounding` | `run_e49adc8d319f` |
|
| 320 |
+
| change | What changed between the earlier and later image? | `change` | `run_aedc59cbcdc9` |
|
| 321 |
+
| change_vqa | Did the coastline advance between the two observations? | `change_vqa` | `run_62ca98d510be` |
|
| 322 |
+
| optical_sar | β¦combining the optical and SAR observations? | `optical_sar` | `run_beacf6aa4e21` |
|
| 323 |
+
| **grounding** | **Where are the built-up areas in this image?** | **`grounding`** | **`run_467ffa406f22`** |
|
| 324 |
+
| **grounding** | **Where is the new airport?** | **`grounding`** | **`run_46980ba55c62`** |
|
| 325 |
+
|
| 326 |
+
The last two are the router-defect queries. Both previously collapsed to `vqa` and answered "River".
|
| 327 |
+
|
| 328 |
+
### Screenshots
|
| 329 |
+
|
| 330 |
+
Eight captures from the post-fix live run (headed browser, 1384Γ855, one upload per case). Each
|
| 331 |
+
panel shows the run identifier, the frozen config hash `78f1e3700da15aa1`, and the evidence list
|
| 332 |
+
returned by the specialist β nothing is mocked.
|
| 333 |
+
|
| 334 |
+
| | |
|
| 335 |
+
|---|---|
|
| 336 |
+
|  |  |
|
| 337 |
+
| **Grounding** β "Where are the built-up areas in this image?" β the fixed router defect (`run_467ffa406f22`, dispatched `grounding`, not `vqa`) | **Optical-SAR** fusion on a real optical/SAR GeoTIFF pair (`run_beacf6aa4e21`, fused class 18) |
|
| 338 |
+
|  |  |
|
| 339 |
+
| **Grounding** β "Where is the new airport?" β second defect query (`run_46980ba55c62`, dispatched `grounding`) | **Grounding** β "Where are the visible buildings in this image?" (`run_e49adc8d319f`) |
|
| 340 |
+
|  |  |
|
| 341 |
+
| **Change** detection on a same-shape temporal pair | **Caption** of a single scene (`run_96281bdfcc08`, calibrated confidence 1.000) |
|
| 342 |
+
|  |  |
|
| 343 |
+
| **VQA** β "What type of terrain dominates this scene?" | **Change-VQA** β "Did the coastline advance between the two observations?" |
|
| 344 |
+
|
| 345 |
+
All eight are reproduced byte-for-byte in the evidence archive (Phase 7) with SHA-256 recorded in
|
| 346 |
+
`RELEASE_MANIFEST.md`.
|
| 347 |
+
|
| 348 |
+
## Installation
|
| 349 |
+
|
| 350 |
+
Python 3.11+ and a CPU are sufficient. No CUDA requirement β device is selected via
|
| 351 |
+
`SATQUERY_DEVICE`; all placement is `.to(device)`, never `.cuda()`.
|
| 352 |
+
|
| 353 |
+
```bash
|
| 354 |
+
git clone https://github.com/Anish-lab-blip/SatQuery-AI
|
| 355 |
+
cd SatQuery-AI
|
| 356 |
+
python -m venv .venv
|
| 357 |
+
source .venv/Scripts/activate # Windows git-bash; use .venv/bin/activate on Linux/macOS
|
| 358 |
+
pip install -r requirements.txt
|
| 359 |
+
```
|
| 360 |
+
|
| 361 |
+
Backbones are fetched from the Hugging Face Hub on first use, pinned by revision in
|
| 362 |
+
`configs/base.yaml`. The frozen config hash is **`78f1e3700da15aa1`** β the loader refuses to run a
|
| 363 |
+
config that violates the recorded invariants (for example `fusion.input_dim == 3*encoder_dim + 12 + 2`).
|
| 364 |
+
|
| 365 |
+
## Local development
|
| 366 |
+
|
| 367 |
+
```bash
|
| 368 |
+
# Inference service, CPU (this is the launcher the Codespace runs)
|
| 369 |
+
PORT=8000 python deploy/codespace/serve.py
|
| 370 |
+
|
| 371 |
+
# Health
|
| 372 |
+
curl localhost:8000/v1/health
|
| 373 |
+
```
|
| 374 |
+
|
| 375 |
+
The frontend is fully static and needs no build step to serve locally:
|
| 376 |
+
|
| 377 |
+
```bash
|
| 378 |
+
python -m http.server 5500 --directory frontend
|
| 379 |
+
```
|
| 380 |
+
|
| 381 |
+
Run the frontend regression suite:
|
| 382 |
+
|
| 383 |
+
```bash
|
| 384 |
+
python -m pytest tests/unit/test_frontend_live_wiring.py -q
|
| 385 |
+
```
|
| 386 |
+
|
| 387 |
+
## Deployment
|
| 388 |
+
|
| 389 |
+
The live topology is **Cloudflare Pages β Render β outbound tunnel β GitHub Codespace**.
|
| 390 |
+
|
| 391 |
+
| Layer | Role | Source |
|
| 392 |
+
|---|---|---|
|
| 393 |
+
| Cloudflare Pages | Static frontend at **https://satquery.pages.dev** | `frontend/` |
|
| 394 |
+
| Render | Orchestrator / API gateway, `/api/*`, CORS, wake flow | `gateway/` |
|
| 395 |
+
| GitHub Codespace | FastAPI inference host, CPU, port 8000 | `app/` |
|
| 396 |
+
| Hugging Face | Model cards, released artifacts, checksums | this release |
|
| 397 |
+
|
| 398 |
+
Deployment sources are **separate repositories** from this release. The wake flow is: Cloudflare β
|
| 399 |
+
Render β start the Codespace if stopped β poll `/v1/health` β surface *"Waking inference engineβ¦"* β
|
| 400 |
+
`POST /infer` β result.
|
| 401 |
+
|
| 402 |
+
### Deployment caveats
|
| 403 |
+
|
| 404 |
+
- **Cold start.** The inference host may be stopped when idle. The first request after a cold start
|
| 405 |
+
can exceed the client timeout while weights are fetched; a retry a few seconds later normally
|
| 406 |
+
succeeds. Warm the stack before any demonstration and confirm
|
| 407 |
+
`GET /api/health` reports `tunnel.agent_connected: true`.
|
| 408 |
+
- **Tunnel gaps.** The tunnel agent can be briefly absent. A request issued during such a gap may hang
|
| 409 |
+
or return HTTP 504. **This is not fixed in production** β a prepared patch
|
| 410 |
+
(`forward_unavailable` 503 / `upstream_timeout` 504 plus a `codespace_name` fix) exists and is
|
| 411 |
+
documented, but it was deliberately not deployed. Root cause: in `auto` transport mode a tunnel
|
| 412 |
+
timeout falls through to the forwarded-port path, which then spends the 120 s wake timeout on an
|
| 413 |
+
HTTP 302 β the observed ~249 s failure.
|
| 414 |
+
- **`codespace_name`** is still reported with a trailing newline by `/api/health` (cosmetic; the wake
|
| 415 |
+
path strips it).
|
| 416 |
+
|
| 417 |
+
## Reproducibility
|
| 418 |
+
|
| 419 |
+
1. **Configuration.** `configs/base.yaml` is the single registry; no magic numbers in Python. Its hash
|
| 420 |
+
is recorded in every execution trace. Frozen hash: `78f1e3700da15aa1`.
|
| 421 |
+
2. **Backbones.** Pinned by `repo_id` + `revision`, never by floating tag:
|
| 422 |
+
`HuggingFaceTB/SmolVLM-500M-Instruct@a7da5b986cb5`,
|
| 423 |
+
`chendelong/RemoteCLIP@bf1d8a3ccf2d`,
|
| 424 |
+
`sentence-transformers/all-MiniLM-L6-v2@1110a243fdf4`,
|
| 425 |
+
`antofuller/CROMA@0dd28e3d633b`.
|
| 426 |
+
3. **Released artifacts.** `models/manifest.json` and `models/checksums.sha256` are **generated from
|
| 427 |
+
the actual files** β never hand-typed. Verify with `sha256sum -c models/checksums.sha256`.
|
| 428 |
+
4. **Splits.** LEVIR-CD-256: train 7120 / val 1024 / test 2048. Grounding: VRSBench n=16159.
|
| 429 |
+
Fusion: held-out test n=4000. Change-VQA: test n=39686. Leakage isolation is by `scene_id`.
|
| 430 |
+
5. **Prompts** are versioned files, frozen before benchmark evaluation.
|
| 431 |
+
6. **Negative results are preserved.** Rejected and open rulings are recorded, not removed.
|
| 432 |
+
|
| 433 |
+
## Known limitations
|
| 434 |
+
|
| 435 |
+
1. **No system-level end-to-end benchmark exists.** Per-specialist metrics are real; a single
|
| 436 |
+
end-to-end number is **NOT RUN**.
|
| 437 |
+
2. **Router accuracy is validation-only** (n=86, corpus-limited). Its **test set was never run**.
|
| 438 |
+
3. **Optical-SAR fusion returns a bare class index** (`class_18`), not a human-readable label. The
|
| 439 |
+
modality accounting in the response confirms the right channels reached the fusion head, but the
|
| 440 |
+
presentation is not user-facing.
|
| 441 |
+
4. **VQA is weak-but-related.** Asked what terrain dominates a scene, it answers "Grassland".
|
| 442 |
+
5. **Fusion macro_F1 is low (0.434)** against 0.931 accuracy β rare classes are poorly handled.
|
| 443 |
+
6. **Grounding IoU is modest** (0.2838 canonical) β useful, not solved.
|
| 444 |
+
7. **Calibration makes ECE slightly worse**, and is retained only because it is part of the frozen
|
| 445 |
+
configuration.
|
| 446 |
+
8. **Router lexical residuals.** *"What is the new runway?"* reads `change` rather than `vqa` (the
|
| 447 |
+
`new`-as-change heuristic fires outside `where` questions), and *"How much built-up area was
|
| 448 |
+
added?"* reads `vqa` (under-trigger). A lexical router cannot cleanly separate "the new X" from
|
| 449 |
+
"what's new"; a trained intent router exists in `artifacts/router/` but is not attached.
|
| 450 |
+
9. **B-07 tunnel gaps are not fixed in production** (see Deployment caveats).
|
| 451 |
+
10. **No license has been selected** for this repository. Until one is, the artifacts carry
|
| 452 |
+
`license: unknown` and no reuse rights should be assumed. This is an open owner decision.
|
| 453 |
+
11. **The Anatomy of a Run page** renders a recorded run whose plate uses the 720Γ720 variant of an
|
| 454 |
+
image analysed at 730Γ730 β identical content, scaled by the canvas, but the "actual analysed
|
| 455 |
+
image" wording is slightly loose.
|
| 456 |
+
|
| 457 |
+
## Links
|
| 458 |
+
|
| 459 |
+
| | |
|
| 460 |
+
|---|---|
|
| 461 |
+
| Live demo | https://satquery.pages.dev |
|
| 462 |
+
| GitHub | https://github.com/Anish-lab-blip/SatQuery-AI |
|
| 463 |
+
| Hugging Face | https://huggingface.co/thundercode/SatQuery |
|
| 464 |
+
|
| 465 |
+
Third-party models this work builds on (pinned, not redistributed):
|
| 466 |
+
|
| 467 |
+
| Model | Revision | Role |
|
| 468 |
+
|---|---|---|
|
| 469 |
+
| [`HuggingFaceTB/SmolVLM-500M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct) | `a7da5b986cb5` | VQA + captioning backbone |
|
| 470 |
+
| [`chendelong/RemoteCLIP`](https://huggingface.co/chendelong/RemoteCLIP) | `bf1d8a3ccf2d` | remote-sensing grounding encoder |
|
| 471 |
+
| [`sentence-transformers/all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | `1110a243fdf4` | router embedding |
|
| 472 |
+
| [`antofuller/CROMA`](https://huggingface.co/antofuller/CROMA) | `0dd28e3d633b` | optical/SAR fusion encoder |
|
| 473 |
+
|
| 474 |
+
Datasets referenced by the evaluations: LEVIR-CD-256 (change), VRSBench (grounding),
|
| 475 |
+
BigEarthNet (optical-SAR fusion, 19 CLC classes). No dataset is redistributed here.
|
| 476 |
+
|
| 477 |
+
## Citation
|
| 478 |
+
|
| 479 |
+
No paper accompanies this release. Until one exists, cite the repository:
|
| 480 |
+
|
| 481 |
+
```bibtex
|
| 482 |
+
@misc{satqueryai2026,
|
| 483 |
+
title = {SatQuery AI: an interactive vision-language assistant for
|
| 484 |
+
multimodal remote-sensing image analysis},
|
| 485 |
+
author = {SatQuery AI contributors},
|
| 486 |
+
year = {2026},
|
| 487 |
+
url = {https://github.com/Anish-lab-blip/SatQuery-AI}
|
| 488 |
+
}
|
| 489 |
+
```
|
| 490 |
+
|
| 491 |
+
## License
|
| 492 |
+
|
| 493 |
+
**Not yet selected.** See limitation 10. Backbone models remain under their own upstream licenses.
|