README.md

---
language: en
license: other
license_name: see-model-card
library_name: pytorch
tags:
  - remote-sensing
  - satellite-imagery
  - earth-observation
  - change-detection
  - visual-grounding
  - image-captioning
  - visual-question-answering
  - optical-sar-fusion
  - sar
  - multimodal
  - lora
  - peft
  - pytorch
base_model:
  - HuggingFaceTB/SmolVLM-500M-Instruct
  - chendelong/RemoteCLIP
  - antofuller/CROMA
  - sentence-transformers/all-MiniLM-L6-v2
pipeline_tag: image-to-text
config_hash: 78f1e3700da15aa1
---

# Model Card — SatQuery AI

SatQuery AI answers natural-language questions about satellite imagery using a **router + specialists**
design. This card documents the **six trained artifacts** released by the project. It is deliberately
explicit about what is measured, what is not, and what was rejected.

> **The six trained artifacts are small modules on top of frozen, publicly-pinned backbones. No
> backbone weights are redistributed by this release** — they are fetched from the Hugging Face Hub at
> run time, pinned by revision.

Machine-readable identities (byte counts and sha256) are in
[`models/manifest.json`](models/manifest.json) and [`models/checksums.sha256`](models/checksums.sha256),
**generated by reading the files** (`release/tools/generate_model_manifest.py`). Where this card and the
generated manifest disagree, the manifest wins — it is computed from disk, this card is written by hand.

**Companion documents** (same depth, same no-fabrication rule): [`docs/MODELS.md`](docs/MODELS.md) ·
[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) · [`docs/TRAINING.md`](docs/TRAINING.md) ·
[`docs/EVALUATION.md`](docs/EVALUATION.md) · [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md) ·
[`docs/RESEARCH_NOTES.md`](docs/RESEARCH_NOTES.md) ·
[`docs/architecture/05-specialists.md`](docs/architecture/05-specialists.md).

---

## Table of contents

1. [How to read this card](#1-how-to-read-this-card)
2. [Overview](#2-overview)
3. [The six artifacts in this release](#3-the-six-artifacts-in-this-release)
4. [Backbone dependencies — frozen, pinned by revision](#4-backbone-dependencies--frozen-pinned-by-revision)
5. [Intended use](#5-intended-use)
6. [Out-of-scope use](#6-out-of-scope-use)
7. [Per-artifact reference](#7-per-artifact-reference)
8. [Full measured-performance table](#8-full-measured-performance-table)
9. [Calibration — a measured negative result](#9-calibration--a-measured-negative-result)
10. [Acceptance status](#10-acceptance-status)
11. [Evaluation gaps](#11-evaluation-gaps)
12. [Limitations](#12-limitations)
13. [Training summary](#13-training-summary)
14. [Provenance and verification](#14-provenance-and-verification)
15. [Licence](#15-licence)
16. [Citation](#16-citation)

---

## 1. How to read this card

**The single most important rule in this document: do not fabricate.** Every byte count, sha256,
hyperparameter and metric below comes from a file that was read, and each one names its source. Where a
fact is not established, this card writes `UNKNOWN — not established from the available evidence`
rather than estimating.

**Status vocabulary.** Every substantive claim carries one of: `IMPLEMENTED` · `VERIFIED` · `MEASURED` ·
`ATTEMPTED` · `NOT RUN` · `BLOCKED` · `DEFERRED` · `REJECTED` · `OPEN` · `RESOLVED` · `CLOSED`.

**The facts most easily stated wrongly, and therefore stated repeatedly:**

| Fact | Correct statement |
|---|---|
| Grounding | measured under **two protocols** (canonical 0.2838 / 0.2198; matched6 0.2566 / 0.1938) **and two decode variants** (head_argmax 0.1215; zero-shot 0.0972). Never quote one alone. |
| Calibration | ECE went **0.013755 → 0.014929 — worse**. Retained only because it is in the frozen config. |
| VLM adapter | metrics **usable** (exact_match 0.963) but status **ACCEPTANCE-REJECTED**. USABLE ≠ ACCEPTED. |
| Optical-SAR | accuracy **0.931** with macro-F1 **0.434161**; ruling **OPEN**. Never accuracy without macro-F1. |
| Change-VQA | **two** test sets: test 0.697626/0.378373 and test2 0.651469/0.372309; ruling **OPEN**. |
| Router | **0.965116 is validation, ungated, n = 86**; the **test split was NOT RUN**. |
| End-to-end benchmark | **does not exist**; no system-level accuracy is claimed. |
| Change | pooled IoU 0.8122 / macro IoU 0.8457 / pooled F1 0.8964 — the **only** `VERIFIED` headline. |

---

## 2. Overview

SatQuery AI is a **router-and-specialists** system: a frozen sentence encoder plus a small trained
adapter classify a query into one of six tasks; a deterministic planner dispatches it to the
appropriate specialist; each specialist returns a structured `ResultEnvelope` carrying evidence and a
confidence value. The design is **CPU-first** and **frozen-backbone** — small modules are trained on top
of pretrained encoders, and no encoder is fine-tuned end to end.

**The six tasks** (`configs/base.yaml` → `router.tasks`):
`vqa` · `caption` · `grounding` · `change` · `optical_sar` · `unsupported`.

**The six trained artifacts** (details in §3):

| Task | What it is | Size |
|---|---|---|
| `change` | STANet-style Siamese change detector (ResNet-18 + PAM) | 63,231,009 B |
| `change_vqa` | two-stage change-reasoning head (`change_vqa_head_v1`) | 5,822,809 B |
| `optical_sar` | CROMA-base fusion head (2318 → 512 → 19) | 14,427,457 B |
| `grounding` | RemoteCLIP grounding head (feature 2048 → hidden 512) | 12,639,041 B |
| `router` | five-head intent adapter over frozen MiniLM | 211,961 B |
| `vlm` | PEFT LoRA adapter on SmolVLM-500M text projections | 34,798,048 B |

Total released weight payload: **131,130,325 bytes (~125 MiB)** (`HF_RELEASE_VERIFICATION.md` §4).

**What is not trained here.** MiniLM, SmolVLM-500M, RemoteCLIP ViT-B/32 and CROMA-base are frozen and
not redistributed. The one nuance is the change detector's ResNet-18, which is *loaded pretrained*
(`pretrained_used: true`) and *trained in-project* as part of the change head — so its weights are part
of the released `change/head.pt`, not a separately-distributed backbone
(`artifacts/change/eval_test/eval_result.json` → `checkpoint_embedded_config`).

**No end-to-end accuracy is claimed anywhere.** The router → specialist → envelope pipeline has never
been scored end to end. What exists is per-specialist metrics on their own training-family splits (§8)
and a behavioural live-validation record that proves the pipeline *runs and routes* — 3 passes × 8
cases, 8/8 each, **24 live runs, 0 mock nodes**, trace fill **94.4444 %** (`docs/BENCHMARKS.md` §5,
`docs/RESEARCH_NOTES.md` §3.3).

---

## 3. The six artifacts in this release

Reproduced from [`models/manifest.json`](models/manifest.json) → `artifacts[*]`, cross-checked against
[`models/checksums.sha256`](models/checksums.sha256). Every artifact carries `status: "PRESENT"` and
`config_hash: "78f1e3700da15aa1"`.

| # | `id` | Task | Kind | File (HF path) | Bytes | sha256 (full) |
|---|---|---|---|---|---|---|
| 1 | `change_head` | `change` | trained head | `change/head.pt` | 63,231,009 | `c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa` |
| 2 | `change_vqa_head` | `change_vqa` | trained head | `change_vqa/head.pt` | 5,822,809 | `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` |
| 3 | `optical_sar_fusion_head` | `optical_sar` | trained head | `optical_sar/head.pt` | 14,427,457 | `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab` |
| 4 | `grounding_head` | `grounding` | trained head | `grounding/head.pt` | 12,639,041 | `93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb` |
| 5 | `router_adapter` | `router` | trained adapter | `router/adapter.pt` | 211,961 | `8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` |
| 6 | `vlm_lora_adapter` | `vlm` | LoRA adapter | `vlm/adapter_model.safetensors` | 34,798,048 | `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` |

The manifest also records each artifact's **original repository path** and its **source metric
artifact**:

| # | `id` | `path` (source repo) | `source_metric_artifact` |
|---|---|---|---|
| 1 | `change_head` | `artifacts/change/levir_change_v001/head.pt` | `artifacts/change/eval_test/eval_result.json` |
| 2 | `change_vqa_head` | `artifacts/change_vqa/run/head.pt` | `artifacts/change_vqa/run/PROMOTION.json` |
| 3 | `optical_sar_fusion_head` | `artifacts/optical_sar/fusion_head_production_v001/head.pt` | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` |
| 4 | `grounding_head` | `artifacts/grounding/remoteclip_grounding_v001/head.pt` | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json` |
| 5 | `router_adapter` | `artifacts/router/router_adapter_v001/adapter.pt` | `artifacts/router/threshold_sweep_val.json` |
| 6 | `vlm_lora_adapter` | `.scratch/phase6_real_adapter/phase6_adapter/adapter_model.safetensors` | `artifacts/vlm/phase6_closure.json` |

**`kind` semantics.** `trained_head` = a module trained in-project on a frozen encoder, loaded via
`torch.load` of a `state_dict` (or the module's own loader). `trained_adapter` = a small classifier over
a frozen sentence encoder's cached embeddings, loaded via `IntentAdapter.from_config_dict` +
`load_state_dict` (`router/adapter.py`). `lora_adapter` = a PEFT LoRA delta attached at load time via
`peft.PeftModel.from_pretrained(model, dir)` (`specialists/vqa/model.py`).

**Two independent cross-checks (not self-consistency).** The manifest is generated by hashing the files
on disk; for two artifacts the computed digest can be compared against a value recorded independently,
at a different time, by a different process:

- **`change_vqa` = `cfae5e43…d63a82a`.** Equals `artifacts/change_vqa/run/PROMOTION.json` →
  `artifact.sha256`, the digest recorded in the Kaggle run record *before* promotion
  (`source.checkpoint_sha256_in_run_record`), and `artifacts/calibration_v001.json` →
  `provenance.checkpoint_sha256` (recorded when the temperature was fitted — a separate step).
  `PROMOTION.json` → `source.hash_agrees_across` records the digest agreeing across
  `model_metadata.json`, `run_record.json` and `hashes.json`, with `byte_identical_to_source: true` and
  `artifact.weights_modified: false`.
- **`vlm` = `07c76a75…a5adf5e`.** Equals `artifacts/vlm/phase6_closure.json` →
  `why_usable_verified.adapter_provenance` → `adapter_verification.json` → `weights_file_sha256`, and the
  adapter's own `ARTIFACT_SHA256SUMS.json`, against which the 14-file directory was verified
  (`manifest_check.clean: true`, 14/14 present).

A third, independent re-download check: `release/tools/hf_verify.py` re-downloads each artifact over
direct HTTPS and hashes the received bytes — **6/6 MATCH, 0 failed**
([`HF_RELEASE_VERIFICATION.md`](HF_RELEASE_VERIFICATION.md) §5).

**On `parameters: null`.** Four of six artifacts record `parameters: null` deliberately — the generator
does not open checkpoints (that would make generation depend on the model code and torch). Counts
measured elsewhere appear in §7 with their source; where a count is not established this card writes
`UNKNOWN — not established from the available evidence`. Training checkpoints are **not** released
artifacts: the VLM adapter's `checkpoint-1500/`/`checkpoint-2000/` are provenance only, and the promoted
adapter is the top-level end-of-training save, not `checkpoint-2000` (§7.6).

---

## 4. Backbone dependencies — frozen, pinned by revision

Backbones are resolved from the Hugging Face Hub on first use, **pinned by revision** — a moving `main`
would make every benchmark number unreproducible.

| Role | Repository | Revision | Size | Measured identity | Notes |
|---|---|---|---|---|---|
| Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | 90.9 MB | 22,713,216 params, 384-dim | tokenizer ceiling **256**; truncation **128** |
| VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | ~1015 MB safetensors | 516,165,824 params (base) | processor `longest_edge` must be pinned (F5-2) |
| Grounding | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`) | `bf1d8a3ccf2d` | 605.2 MB | 151,277,313 params; width **768**, projected **512** | patch 32; 7×7 tokens at 224 |
| Optical-SAR | `antofuller/CROMA` (`CROMA_base.pt`) | `0dd28e3d633b` | 777.6 MB (777,563,846 B) | 194,365,440 params; `encoder_dim` 768 | resolution 120; asymmetric `s1_depth=6`, `s2_depth=12` |
| Change encoder | — (torchvision) | — | — | ResNet-18, `IMAGENET1K_V1` | `pretrained_used: true` in the artifact |

All pins are declared in `configs/base.yaml` under the `router:`, `vlm:`, `grounding:` and `croma:`
blocks and are validated at load time by `core/config.py`. Backbone licence terms are each repository's
own — see §15. **Backbones are not redistributed here.**

---

## 5. Intended use

- **Research and demonstration** of a modular, CPU-first remote-sensing question-answering system.
- **Routing and dispatch** of natural-language queries to the appropriate specialist, using the
  `router` adapter over frozen MiniLM embeddings.
- **Reproducible evaluation** of each specialist on its own documented split, using the released
  artifacts and the frozen config hash `78f1e3700da15aa1`.
- **Teaching and ablation**: the six artifacts are small and individually inspectable; the frozen-backbone
  design makes each head a self-contained experiment.

The artifacts are intended to be used **with their pinned backbones** (§4), which the consumer must
fetch separately.

---

## 6. Out-of-scope use

- **Safety-, legal- or life-critical decisions.** No accuracy, calibration or robustness guarantee is
  offered. Grounding boxes are **image-relative, not geodetic** — no geolocation accuracy
  (`docs/LIMITATIONS.md` §7).
- **Operational geospatial production** without independent validation.
- **Any use of the VLM adapter as a production model** — it is `ACCEPTANCE-REJECTED` (§10); the deployed
  caption/VQA path uses the **unadapted** model.
- **Treating per-specialist metrics as system-level accuracy.** No end-to-end benchmark exists (§11).
- **Any claim that these artifacts generalise beyond their training-family test splits** — cross-dataset
  generalisation is `NOT RUN`.
- **Redistribution of the backbones.** This release contains no backbone weights.

---

## 7. Per-artifact reference

Each subsection gives architecture, hyperparameters (from `configs/base.yaml` unless noted), training
data, evaluation protocol, measured numbers (with source artifact and key path), acceptance status, and
limitations.

### 7.0 Enforced configuration invariants, with arithmetic

The frozen registry is `configs/base.yaml`; the project rule is **"no magic numbers anywhere in Python;
everything tunable lives here"**, and `core/config.py` loads, validates and hashes every value. Several
values are *enforced* — a mismatch is a load-time error, not a comment:

| Invariant | Arithmetic / rule | Why it is enforced |
|---|---|---|
| Fusion input width | `3 × 768 + 12 + 2 = 2318` | `core/config.py` recomputes it and `fusion_head.py` recomputes it **again**, refusing to build on mismatch — a config edit cannot silently reshape the first `Linear` |
| Grounding head feature width | `4 × 512 = 2048` | `core/config.py` rejects any other value and `specialists/grounding/remoteclip.py` asserts it against the real model — a mismatch is a **silent** shape error otherwise |
| CROMA resolution | `image_resolution % 8 == 0`; native `120` → `225` patches | required by CROMA (finding C-7) |
| Router truncation | `max_length ≤ 256` | MiniLM tokenizer ceiling; truncating above it is a silent no-op (F4-1) |
| VLM processor | `processor_longest_edge ≤ image.tile_size` | otherwise the processor upscales and splits a tile ~17× (F5-2) |
| Change tile | `256`; `tile_overlap: 0` | STANet-style detector; LEVIR-CD-256 |
| Frozen config hash | `Config.hash = sha256(base.yaml)[:16] = 78f1e3700da15aa1` | every artifact records it; a config edit detaches the numbers from their configuration |

The frozen hash is verified untouched by `test_the_frozen_config_hash_has_not_moved`
(`docs/OWNER_DECISIONS_2026-09-23.md`, cross-cutting rule 4). New defaults live in **code**, not in the
registry — which is why the grounding head's default path is `DEFAULT_HEAD_PATH` in code rather than a
`base.yaml` key (owner decision D-4).

---

### 7.1 `change` — STANet-style Siamese change detector

**Kind:** trained head · **File:** `change/head.pt` · **Bytes:** 63,231,009 · **sha256:**
`c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa`

**Architecture.** A STANet-style Siamese detector: a shared ResNet-18 encoder (`SharedResNetEncoder`), a
`DifferenceFusion` module per stage, **PAM** spatial self-attention (`SpatialAttention`, `sa_mode: PAM`;
BAM is the alternative), a three-stage decoder (`dec3 → dec2 → dec1`) with a final upsample and a 1×1
convolution head to one change logit. Source: `specialists/change/stanet.py`. Manifest architecture
string: `"STANet-style Siamese change detector (ResNet-18 + PAM)"`.

**Hyperparameters** (`base.yaml` → `change:`; artifact `checkpoint_embedded_config`):

| Parameter | Value | | Parameter | Value |
|---|---|---|---|---|
| `tile_size` / `tile_overlap` | 256 / 0 | | `threshold` | 0.50 |
| `min_component_pixels` | 32 | | `encoder` | `resnet18` |
| `encoder_channels` | `[64,128,256,512]` | | `width` | 128 |
| `sa_mode` | `PAM` | | `pretrained` | `true` |
| `frozen_encoder` | `false` | | `attention_budget_bytes` | 268,435,456 |
| `learning_rate` / `batch_size` | 0.001 / 8 | | `bce_weight` / `dice_weight` | 0.5 / 0.5 |

**Training data.** LEVIR-CD-256, split `train 7,120 / val 1,024 / test 2,048` (`base.yaml` →
`change.levir_split`), matching the published LEVIR-CD counts exactly
(`docs/OWNER_DECISIONS_2026-09-23.md` D-11). Trained on GPU (eval artifact: `device: cuda`,
torch 2.10.0+cu128, python 3.12.13).

**Evaluation protocol.** Held-out `test` split, n = 2048, threshold 0.50, tile 256, no overlap. Source:
`artifacts/change/eval_test/eval_result.json`. The artifact checks the checkpoint's embedded config
against the frozen hash (`checkpoint_config_hash_checked: true`, `config_drift: false`).

**Measured numbers:**

| Metric | Value | Key path |
|---|---|---|
| **pooled IoU** | **0.8122** | `metrics.pooled.iou` |
| **macro IoU** | **0.8457** | `metrics.macro.miou` |
| **pooled F1** | **0.8964** | `metrics.pooled.f1` |
| pooled `miou` / precision / recall | 0.9007 / 0.9195 / 0.8745 | `metrics.pooled.*` |
| macro F1 / `iou` / precision / recall | 0.7962 / 0.7180 / 0.8506 / 0.7757 | `metrics.macro.*` |
| confusion | tp 5,978,997 · fp 523,658 · fn 858,407 · tn 126,856,666 | `metrics.pooled.*` |
| n pixels | 134,217,728 | `metrics.pooled.n_pixels` |
| images with change | 935 / 2,048 | `n_images_with_change` |
| mean change fraction | 0.0509 (p50 0.0, p90 0.197205, max 0.684937) | `metrics.mean_change_fraction`, `change_fraction_quantiles` |
| wall time | 55.359 s | `metrics.seconds` |

**Acceptance status:** **VERIFIED** and accepted (shipped). This is the **only** headline metric in the
project carrying the `VERIFIED` tag — measured against a single, immutable public test split with a
frozen threshold (`docs/BENCHMARKS.md` §1.2).

**Limitations.** Pooled and macro figures diverge (IoU 0.8122 vs 0.8457; F1 0.8964 vs 0.7962), and the
corpus is heavily zero-change (`p50` change fraction 0.0; only 935 of 2,048 images contain change). No
cross-dataset evaluation was run.

---

### 7.2 `change_vqa` — change question answering head

**Kind:** trained head · **File:** `change_vqa/head.pt` · **Bytes:** 5,822,809 · **sha256:**
`cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` · **Parameters:** 1,453,912

**Architecture.** `change_vqa_head_v1` — a **two-stage** reasoning head, not a generative decoder
(`training/change_vqa/model.py`). Stage 1 maps the change representation to a **class-wise change
estimate** — 6 magnitudes + 6 signed deltas + 1 global fraction = **13 outputs**
(`N_ESTIMATOR_OUTPUTS = 2 × N_CHANGE_CLASSES + 1`), supervised by `label1`/`label2`. Stage 2
concatenates that estimate with a question encoding and predicts one of **19 answers** (`N_ANSWERS`).
The estimator's outputs are *also* emitted as evidence, so an answer arrives with its own audit trail.
Modules: an `estimator` MLP (`change_feature_dim → 256 → 13`), a `change_trunk`, a `question_trunk`
(text feature + question-type embedding + temporal embedding) and an `answer_head`
(`fused → 512 → 256 → 19`).

**Hyperparameters** (module constants; `base.yaml` → `training:` where applicable):

| Parameter | Value | | Parameter | Value |
|---|---|---|---|---|
| `ARCHITECTURE_VERSION` | `change_vqa_head_v1` | | `trunk_dim` / `text_dim` | 512 / 256 |
| `dropout` | 0.10 | | `qtype_embed_dim` / `temporal_embed_dim` | 32 / 8 |
| estimator outputs | 13 (2 × 6 + 1) | | answer space | 19 |
| `seed` | 42 | | `epoch_selected` | 8 (on Val answer accuracy) |
| `stop_reason` | `early_stopping` | | | |

**Training data.** CDVQA (`dataset_id: cdvqa`), feature specs `change_feat_v1`,
`change_cache_spec: c801326f85a185f8`, `text_cache_spec: d2801ea1a314354a`,
`preprocessing_version: change_vqa_preproc_v1`. The head's change features are backed by the **frozen**
STANet change detector — `frozen_dependency.path: artifacts/change/levir_change_v001/head.pt`, sha256
`c5ef31277b67aa01…`, `verified_byte_exact_vs_local: true` (the *same* artifact as §7.1). Trained on an
external GPU (Kaggle) — see §13.

**Evaluation protocol.** Two held-out test sets, `test` (n = 39,686) and `test2` (n = 31,036).
`epoch_selected` was chosen on **Val answer accuracy = 0.700018**; `PROMOTION.json` records 93 checks
passed, 0 failed, 0 unverified. Source: `artifacts/change_vqa/run/PROMOTION.json` → `verification`.

**Measured numbers:**

| Metric | `test` | `test2` |
|---|---|---|
| **accuracy** | **0.697626367** | **0.651469262** |
| **macro F1** | **0.378373275** | **0.372308516** |
| global-majority baseline | 0.311546 | 0.178728 |
| n scored | 39,686 | 31,036 |

`mask_gain: 0.0`. `metric_ruling: "OPEN — the plan leaves the accuracy/macro-F1 interpretation
owner-gated. No official aggregate metric is asserted here."`

**Acceptance status:** MEASURED on two test sets; **ruling OPEN**. `PROMOTION.json` is explicit that
promotion "records provenance and wires the serving path. It does not itself confer VERIFIED status."

**Limitations.** The wide accuracy–macro-F1 gap (0.697626 vs 0.378373) is the signature of class
imbalance: accuracy is dominated by frequent answers while macro-F1 exposes weak rare-class performance
(`docs/LIMITATIONS.md` §1.4). Confidence at this head is **raw, not calibrated** (`method` reads
`"uncalibrated"`). The two test sets disagree (0.697626 vs 0.651469), so quoting one alone is selective.

---

### 7.3 `optical_sar` — CROMA-base fusion head

**Kind:** trained head (production) · **File:** `optical_sar/head.pt` · **Bytes:** 14,427,457 ·
**sha256:** `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab`

**Architecture.** A fusion head over **frozen CROMA-base** features. CROMA emits three 768-d GAP vectors
per sample (`optical_GAP`, `SAR_GAP`, `joint_GAP`); the head concatenates them with the availability
masks — `optical_mask (B,12)` and `sar_mask (B,2)` — into a **(B, 2318)** tensor (`3×768 + 12 + 2`), then
`LayerNorm → Linear(2318 → 512) → GELU → Dropout(0.2) → Linear(512 → 19)`. Source:
`specialists/optical_sar/fusion_head.py`. The **availability mask is consumed by the head, not by
CROMA** (finding C-1): handing CROMA the mask would invite it to reconstruct missing channels — the
fabrication the sensor adapter exists to prevent.

**Hyperparameters** (`base.yaml` → `croma:` and `fusion:`):

| Parameter | Value | | Parameter | Value |
|---|---|---|---|---|
| `croma.checkpoint_file` | `CROMA_base.pt` (rev `0dd28e3d633b`) | | `croma.image_resolution` | 120 (`% 8 == 0`) |
| `croma.encoder_dim` | 768 | | `croma.optical_channels` / `sar_channels` | 12 / 2 |
| `croma.modalities_used` | `[optical, sar, joint]` | | `fusion.input_dim` | 2318 |
| `fusion.hidden_dim` | 512 | | `fusion.dropout` | 0.2 |
| `fusion.num_classes` | 19 (BigEarthNet CLC) | | channel/band dropout | **mandatory** (freeze §2.5) |

**Training data.** reBEN / BigEarthNet-S1 (`data/bigearthnet_v2/`, 480,038 rows in `metadata.parquet`;
`docs/OWNER_DECISIONS_2026-09-23.md` D-11). The extraction used the **`require_single_label` policy**
(`n_skipped_by_policy: 0`), which preserves the frozen single-label 19-class softmax but **changes the
evaluation population** (see limitations). The A/B arm decision was made separately on
`best_val_accuracy` — **A 0.837100 vs B 0.839100**, floor 0.0285 → **Arm A retained** (owner ruling
R-14; `docs/PHASE12_115_METRIC_COMPUTED.md` §5).

**Evaluation protocol.** The pre-registered 11.5 metric: fusion-head accuracy and macro-F1 over the
19-class label space on the held-out `test` split, n = 4,000, cache arm A
(`docs/PHASE14_CROMA_NORMALISATION_CHANGE.md` §4). Computed by a **separate, later, read-only** tool
(`scripts/eval_fusion_115.py`); the trainer deliberately never opens the test split
(`pre_registered_metric_computed = false` in every run record). Source:
`artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`.

**Measured numbers:**

| Metric | Value | Key path |
|---|---|---|
| **accuracy** | **0.931** | `accuracy` |
| **macro F1** | **0.434161** | `macro_f1` |
| loss | 0.254592 | `loss` |
| n scored / classes | 4,000 / 19 | `n_scored` / `num_classes` |
| classes present | `[0,2,3,4,5,6,7,8,9,10,12,13,17,18]` | `classes_present` |
| classes absent | `[1,11,14,15,16]` | `classes_absent` |
| macro-F1 denominator | all 19 slots (absent classes contribute 0.0) | `macro_f1_denominator` |
| present-only macro-F1 (diagnostic) | 0.589218 | `docs/PHASE12_115_METRIC_COMPUTED.md` §3.5 |

The majority class holds 2,264 / 4,000 = **0.566**, so 0.931 is not a constant predictor. Per-class F1
(`_per_class_f1`) shows a wide spread: one class is perfect (1.000), while classes **5** and **6** are
**present but score 0.000** — genuine per-class failures, not absent-class artifacts. The median of the
14 present classes is **0.6857** against an accuracy of 0.931 — the signature of prediction dominated by
frequent classes (`docs/PHASE12_115_METRIC_COMPUTED.md` §3.4–§3.5).

**Acceptance status:** MEASURED; **ruling OPEN**. Whether 0.931/0.434 constitutes a Phase 12 *pass* is
the **metric-of-record ruling**, which "has not been made, and it is not engineering's to make." The
artifact's own `is_deciding_statistic: false` and advisory text state that it "selects no head, ranks
nothing and compares no arms."

**Limitations.** (1) **Never quote accuracy alone** — 0.931 travels with macro-F1 0.434161. (2) The
metric describes a **single-label subset**, not multi-label reBEN: single-label patches are 17.57 % of
the corpus (96,537 / 549,488), and under this policy the rarest class survives as **1 patch** (a
59,204 : 1 imbalance). It **may not** be presented as a multi-label BigEarthNet/reBEN result, nor as
comparable to published BigEarthNet numbers, nor as a statement about all 19 classes — 5 have no test
samples here. (3) The live service returns a **bare class index** (`class_18`), not a CLC label
(`docs/LIMITATIONS.md` §1.6).

---

### 7.4 `grounding` — RemoteCLIP grounding head

**Kind:** trained head · **File:** `grounding/head.pt` · **Bytes:** 12,639,041 · **sha256:**
`93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb` · **Parameters:** 1,052,677
(`docs/OWNER_DECISIONS_2026-09-23.md` D-4, measured against the real checkpoint).

**Architecture.** A text-conditioned per-cell box regressor over **frozen RemoteCLIP ViT-B/32** tokens.
At 224 px the patch grid is **7×7 = 49** tokens of projected dim **512**; the text embedding (512) is
broadcast to every cell, and each cell's feature is `concat([patch, text, patch*text, global_pool]) =
4 × 512 = 2048`. The head is `Linear(2048 → 512) → LayerNorm → Dropout(0.10) → Linear(512 → 5)`,
emitting `[tx, ty, tw, th, obj]` per cell. Boxes are **cell-relative** (YOLO-style), and exactly **one
cell per target** is positive — the one containing the ground-truth box centre. Source:
`specialists/grounding/head.py`.

**Hyperparameters** (`base.yaml` → `grounding:`, `grounding_head:`, `grounding_training:`):

| Parameter | Value | | Parameter | Value |
|---|---|---|---|---|
| `image_size` | 224 (`resolution_frozen: true`) | | `model_name` | `ViT-B-32` |
| `encoder_projected_dim` | 512 (width 768 → projected 512, P7-1) | | `nms_iou` | 0.50 |
| `max_candidates` | 20 | | `confidence_threshold` | 0.40 |
| `benchmark_box_scale` | 100.0 (VRSBench 0–100 → stored 0–1) | | `head.feature_dim` | 2048 |
| `head.hidden_dim` | 512 | | `head.dropout` | 0.10 |
| `head.positive_confidence_weight` | 20.0 (1 positive in 49) | | `head.decode` | `cell_relative` |
| training lr / batch / epochs | 1e-4 / 16 / 20 | | training wd / warmup / grad_clip | 1e-4 / 0.05 / 1.0 |
| training `val_fraction` | 0.10 | | loss weights box/giou/conf | 0.5 / 0.3 / 0.2 |

**Training data.** VRSBench (`training/data/vrsbench/`), 16,159 eval records, all images present
(`docs/OWNER_DECISIONS_2026-09-23.md` D-11). Resolution frozen at **224** by a pre-registered decision
(see protocol).

**Evaluation protocol.** Full VRSBench eval split, **16,159 / 16,159 records**, resolution 224, CPU
(canonical and matched6 artifacts record `device: cpu`, torch 2.14.0+cpu). Grounding is reported under
**two protocols** — *canonical* (config default `top_k = 20`) and *matched6* (`top_k = 6`, matching the
zero-shot baseline's mean 5.99 candidates) — and **two decode variants** — `head_threshold` (score
threshold 0.40) and `head_argmax`. `head_decode`: `nms_iou 0.5`, `score_threshold 0.4`.

**Measured numbers** (canonical: `…/eval_result_canonical.json`; matched6: `…/eval_result_matched6.json`):

| Protocol / decode | mean best IoU | recall@0.10 | recall@0.25 | recall@0.50 |
|---|---|---|---|---|
| **canonical** `head_threshold` | **0.2838** | 0.6882 | 0.5047 | **0.2198** |
| **canonical** `head_argmax` | **0.1215** | 0.3183 | 0.2088 | 0.0795 |
| **canonical** `zero_shot_matched` | **0.0972** | 0.3298 | 0.1188 | 0.0234 |
| **matched6** `head_threshold` | **0.2566** | 0.6315 | 0.4545 | **0.1938** |
| **matched6** `head_argmax` | **0.1215** | 0.3183 | 0.2088 | 0.0795 |
| **matched6** `zero_shot_matched` | **0.0972** | 0.3298 | 0.1188 | 0.0234 |

Latency: `head_threshold` 2.205 ms/image (canonical) / 2.158 (matched6); `head_argmax` 0.655 / 0.652;
`zero_shot_matched` 17.9 s / 15.5 s total. The zero-shot decode is
`threshold_box_plus_local_maxima`, `delta 0.02`, `top_k 5`, mean 5.99 candidates/image. The artifact's
`phase7_reference` records the zero-shot floor `mean_best_iou 0.0972`, `recall_at_0.50 0.0234`
(`docs/PHASE7_RESOLUTION_DECISION.md`).

**The 224-vs-448 decision (pre-registered, then confirmed).** The rule was fixed *before* the result was
seen: *448 wins if Recall@0.5 improves by ≥ 0.05 absolute OR mean best IoU improves by ≥ 0.05 absolute;
224 wins otherwise.* Result: **224 WINS** — mean best IoU gain −0.0147, recall@0.5 gain −0.0022, at
1.59× the latency. Paired over the identical 16,159 samples: mean paired diff −0.0147, 95 % CI
[−0.0160, −0.0134], **t = −22.63**; 448 better on 8.5 %, worse on **20.9 %**. The artifact records
`rule_changed_since_preregistration: false` (`docs/PHASE7_RESOLUTION_DECISION.md`).

**Acceptance status:** MEASURED under two protocols; shipped. The trained head is the **production
default** (owner decision D-4); zero-shot is an **explicit, labelled fallback**, and the system must
never silently claim `trained` while running zero-shot.

**Limitations.** Absolute IoU is low (0.2838 canonical / 0.2566 matched6) — the head clearly beats the
zero-shot baseline (0.0972) but 0.28 is not "solved". The number is **protocol-sensitive**: an absolute
value is meaningless without its protocol and decode variant. `head_argmax` (0.1215) is **not**
apples-to-apples with the multi-box baseline (mean best IoU is a max over predictions, so 1 box vs ~6
flatters the head). Boxes are image-relative, not geodetic. 448 was rejected at the *zero-shot* level;
whether a *learned* head has the same resolution sensitivity is `UNKNOWN — not established from the
available evidence` (`docs/PHASE7_RESOLUTION_DECISION.md`).

---

### 7.5 `router` — intent adapter over frozen MiniLM

**Kind:** trained adapter · **File:** `router/adapter.pt` · **Bytes:** 211,961 · **sha256:**
`8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` · **Parameters:** ~50,822

**Architecture.** The only trainable part of the router (`router/adapter.py`, `IntentAdapter`):

```
embedding (384) → LayerNorm → Linear(384 → 128) → GELU → Dropout(0.10)
  ├── task_head      Linear(128 → 6)   # vqa/caption/grounding/change/optical_sar/unsupported
  ├── modality_head  Linear(128 → 4)
  ├── temporal_head  Linear(128 → 1)   # logit; P(yes) = sigmoid(logit)
  ├── spatial_head   Linear(128 → 1)
  └── language_head  Linear(128 → 1)
```

Heads are initialised with small-std weights (`std 0.02`, zero bias) so the initial sigmoid sits near
0.5 and the binary heads do not start saturated. The adapter does **not** back-propagate into MiniLM.

**Hyperparameters** (`base.yaml` → `router:` and `router.training:`):

| Parameter | Value | | Parameter | Value |
|---|---|---|---|---|
| `model` | `all-MiniLM-L6-v2` (rev `1110a243fdf4`) | | `max_length` | 128 (ceiling is 256) |
| `embedding_dim` | 384 | | `hidden_dim` | 128 |
| `dropout` | 0.10 | | `num_tasks` | 6 |
| `confidence_threshold` | 0.70 | | `epochs` / `batch_size` | 60 / 64 |
| `learning_rate` / `weight_decay` | 0.001 / 0.01 | | loss weights task/modality/binary | 1.0 / 0.3 / 0.5 |
| `val_ratio` | 0.15 | | `hard_negatives_to_test` | `true` |

**Training data.** A synthetic query corpus: **576** queries in **54** groups
(`artifacts/router/threshold_sweep_val.json` → `corpus_total`, `corpus_groups`), split
`train 410 / val 86 / test 80`. Splits are **by group** (template / hard-negative family), never by
example, so template variants cannot leak across the boundary (F4-3). Hard-negative families are placed
in the **test** split so their accuracy measures generalisation, not memorisation. The encoder is frozen,
so embeddings are cached and the adapter trains on cached vectors — measured on CPU: **20 epochs over
4,096 × 384 in 0.28 s** (F4-2).

**Evaluation protocol.** A **validation-only threshold sweep** over 50 thresholds `0.50 … 0.99`, on val
**n = 86**, `select_by: covered_accuracy`. Source: `artifacts/router/threshold_sweep_val.json`. The test
split was **NOT touched** (`test_split_touched: false`, `n_test_examples_scored: 0`).

**Measured numbers:**

| Metric | Value | Key path |
|---|---|---|
| **overall ungated accuracy** | **0.965116** | `overall_ungated_accuracy` |
| n val | 86 | `n_val` |
| corpus total / groups | 576 / 54 | `corpus_total` / `corpus_groups` |
| split sizes | train 410 / val 86 / test 80 | `split_sizes` |
| shipped threshold | 0.70 | `shipped_threshold` |
| shipped row (thr 0.70) | coverage 0.848837 · covered acc 0.972603 · fallback 0.151163 · n_covered 73 | `shipped_row` |
| selected row (thr 0.76) | coverage 0.790698 · covered acc 1.0 · fallback 0.209302 · n_covered 68 | `selected` |
| val per-task support | caption 8 · change 20 · grounding 14 · optical_sar 10 · unsupported 19 · vqa 15 | `val_task_counts` |
| hard negatives in val | 0 | `hard_negatives_in_val` |
| adapter config hash | `615478910dc266bf` | `adapter_config_hash` |
| encoder | 22,713,216 params, `max_length` 128, rev `1110a243fdf4` | `adapter_encoder` |

**Acceptance status:** MEASURED (val only); shipped; **test split NOT RUN**.

**Limitations.** The artifact is explicit that this is **not** a calibration and **not** a test result:
*"corpus-limited: val n=86 vs plan >=500. This is NOT a calibration — the corpus is synthetic and too
small (min per-class support 8, caption) and val carries 0 hard negatives (hn_* families are held out to
TEST by design). Selecting a threshold here yields a justified default, not a calibrated value."*
`plan_min_val_queries: 500` and `plan_min_hard_negatives: 100` are both unmet. The number is
**ungated** accuracy, and the router has known residuals — e.g. *"What is the new runway?"* reads
`change`, not `vqa` (`docs/LIMITATIONS.md` §2).

---

### 7.6 `vlm` — SmolVLM LoRA adapter (USABLE_VERIFIED, ACCEPTANCE-REJECTED)

**Kind:** LoRA adapter (PEFT) · **File:** `vlm/adapter_model.safetensors` · **Bytes:** 34,798,048 ·
**sha256:** `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` · **Trainable params:**
8,683,520 (1.6823 % of the 516,165,824-param base)

**Architecture.** A **PEFT LoRA** adapter (`r = 16`, `alpha = 32`, `dropout = 0.05`) on the
**text-model projections** of frozen `HuggingFaceTB/SmolVLM-500M-Instruct` (rev `a7da5b986cb5`). Target
modules: `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` across **224** modules.
`trainable_subtrees` is exactly `{"model.text_model": 8683520}` — the **vision tower was untouched**
(86,433,024 frozen) and the connector (11,796,480) is frozen too. Precision: **fp16**.

**Hyperparameters** (`base.yaml` → `training:`, `vlm:`; `phase6_closure.json`):

| Parameter | Value | | Parameter | Value |
|---|---|---|---|---|
| `checkpoint` | `SmolVLM-500M-Instruct` (rev `a7da5b986cb5`) | | `lora_rank`/`alpha`/`dropout` | 16 / 32 / 0.05 |
| LoRA target modules | 224 | | `precision` | fp16 (T4 is SM 7.5 → not bf16; C-6) |
| `vlm_batch_size` / `grad_accum` | 2 / 8 | | `vlm_learning_rate` / `vlm_epochs` | 2e-4 / 1 |
| `weight_decay` / `warmup_ratio` | 0.01 / 0.05 | | `gradient_checkpointing` | `true` |
| `processor_longest_edge` | 512 (default 2048 splits a tile into 17 sub-images — F5-2) | | `do_sample` / `temperature` | `false` / 0.0 |
| `max_new_tokens` | 128 | | `max_images_per_call` / `seed` | 1 / 42 |

**Training data.** BigEarthNet-derived presence questions (`kind: bigearthnet_smolvlm_lora`). Trained on
an external GPU (Kaggle, T4) — see §13. The adapter directory holds **14 files** verified against its
own `ARTIFACT_SHA256SUMS.json`.

**Evaluation protocol.** A frozen **1,000-question test subset** (`available_per_split {val: 6750,
test: 7772}`, subset n = 1000, 19 classes summing to 1000). The pre-registered acceptance rule is
**v002**: V1 requires aggregate test delta ≥ +5.00 pp; V2 (a per-class guardrail) fails a class with
`n ≥ 20` questions iff it **both** lost ≥ 4 questions **and** has `z ≥ 1.96`. Source:
`artifacts/vlm/phase6_closure.json` → `why_acceptance_rejected`, `why_usable_verified`.

**Measured numbers:**

| Metric | Value | Key path |
|---|---|---|
| **exact_match** | **0.963** | `why_usable_verified.adapted_test.exact_match` |
| **F1** | **0.96432** | `why_usable_verified.adapted_test.f1` |
| precision / recall | 0.963391 / 0.965251 | `…adapted_test.precision` / `.recall` |
| confusion | tp 500 · fp 19 · tn 463 · fn 18 | `…adapted_test.confusion` |
| n | 1,000 | `…adapted_test.n` |
| aggregate test delta | **+49.50 pp** (46.80 → 96.30) | `why_usable_verified.aggregate_test_delta_pp` |

**Why it is *usable and verified*.** Gate D reproduced Run 1's adapted-test control **exactly**
(`exact_match 0.963`, `f1 0.9643201542912246`, identical confusion), proving the local artifact *is*
Run 1's adapter and that CPU/fp32 reproduces the Kaggle T4 endpoint. Gate A″ proved subset identity
without a model. The 14-file manifest check is **clean** (0 missing, 0 mismatched, 0 extra) and the
adapter **loads through the production path** (`PeftModel.from_pretrained`).

**Why it is *acceptance-rejected*.** V1 **passes** (+49.50 pp ≥ +5.00), but V2 **fails**: class
**Mixed forest** (`n = 33`) goes `100.00 → 87.8788 pp`, a drop of **12.1212 pp**, `lost_questions 4`,
`z 2.1335` — failing **both** halves of v002. Per item V, a complete run that fails V2 is `REJECTED`.
The rejection is narrow (1 of 19 classes fails; 11 improved, 5 held) and is **not a split artefact** —
the same class also degraded on val in Run 1 (drop 6.4516 pp, n = 31). Residual risk, reported not
resolved: the verdict rests on 4 questions in one class of 33, the unfloored minimum-size exposure
recorded at `PHASE6_AUDIT_AND_CONTRACT.md` §8.6.

**Acceptance status:** **USABLE_VERIFIED** and **ACCEPTANCE-REJECTED** — both true, answering different
questions. **`USABLE_VERIFIED` ≠ `ACCEPTANCE-ACCEPTED`.** The deployed caption/VQA path uses the
**unadapted** model; the adapter is enabled only via the `SATQUERY_VLM_ADAPTER` environment variable
(`specialists/vqa/model.py` → `ADAPTER_ENV_VAR`).

**Limitations and traps.** (1) The adapter is **not** accepted for production use. (2) The
`Mixed forest` regression is **not resolved**. (3) `adapter_sha256` names **two different values** and
they are not interchangeable — a **tree hash** over the weight map (`5c6b8631…`, from
`training/vlm/artifact.py`) versus the **file** sha256 of `adapter_model.safetensors` (`07c76a75…`, from
`specialists/vqa/model.py::_adapter_sha256`); comparing one against the other produces a false
"artifact was altered" conclusion. (4) The promoted adapter is **not** `checkpoint-2000` — the three
weight files have three distinct digests (top-level `07c76a75…`, `checkpoint-1500` `7273588e…`,
`checkpoint-2000` `bf249943…`). (5) The adapter's canonical path is under `.scratch/`; it is
**reconstructible** from `phase6_realbundle.zip` and verified against the two digests above.

---

## 8. Full measured-performance table

Every row names its source artifact and the exact key path. The `n` and `split` columns are part of the
claim, not decoration: a metric without its population is not a result. All 20 numeric claims are
checked against these files by [`tools/verify_readme_metrics.py`](tools/verify_readme_metrics.py); its
output (`ALL CLAIMS VERIFIED`) is committed as
[`tools/readme_metrics_report.txt`](tools/readme_metrics_report.txt).

| Capability | Metric | Value | Split / protocol | n | Source → key path | Status |
|---|---|---|---|---|---|---|
| Change | pooled IoU | **0.8122** | LEVIR-CD-256 test, thr 0.50 | 2,048 | `change/eval_test/eval_result.json` → `metrics.pooled.iou` | **VERIFIED** |
| Change | macro IoU | **0.8457** | same | 2,048 | `…` → `metrics.macro.miou` | **VERIFIED** |
| Change | pooled F1 | **0.8964** | same | 2,048 | `…` → `metrics.pooled.f1` | **VERIFIED** |
| Grounding | mean best IoU | **0.2838** | VRSBench canonical (thr, top_k 20) | 16,159 | `grounding/…/eval_result_canonical.json` → `results.head_threshold.mean_best_iou` | MEASURED (2 protocols) |
| Grounding | recall@0.5 | **0.2198** | canonical | 16,159 | `…canonical.json` → `results.head_threshold.recall.0.50` | MEASURED (2 protocols) |
| Grounding | mean best IoU | **0.2566** | VRSBench matched6 (thr, top_k 6) | 16,159 | `…matched6.json` → `results.head_threshold.mean_best_iou` | MEASURED (2 protocols) |
| Grounding | recall@0.5 | **0.1938** | matched6 | 16,159 | `…matched6.json` → `results.head_threshold.recall.0.50` | MEASURED (2 protocols) |
| Grounding | head-argmax IoU | **0.1215** | canonical (argmax) | 16,159 | `…canonical.json` → `results.head_argmax.mean_best_iou` | MEASURED |
| Grounding | zero-shot baseline IoU | **0.0972** | canonical (no head) | 16,159 | `…canonical.json` → `results.zero_shot_matched.mean_best_iou` | MEASURED (baseline) |
| Optical-SAR | accuracy | **0.931** | held-out test, 19 classes | 4,000 | `optical_sar/…/pre_registered_115_metric.json` → `accuracy` | MEASURED, ruling **OPEN** |
| Optical-SAR | macro F1 | **0.434161** | same | 4,000 | `…` → `macro_f1` | MEASURED, ruling **OPEN** |
| Change-VQA | accuracy | **0.697626** | `test` | 39,686 | `change_vqa/run/PROMOTION.json` → `verification.test_accuracy` | MEASURED, ruling **OPEN** |
| Change-VQA | macro F1 | **0.378373** | `test` | 39,686 | `…PROMOTION.json` → `verification.test_macro_f1` | MEASURED, ruling **OPEN** |
| Change-VQA | accuracy (2nd set) | **0.651469** | `test2` | 31,036 | `…PROMOTION.json` → `verification.test2_accuracy` | MEASURED, ruling **OPEN** |
| Change-VQA | macro F1 (2nd set) | **0.372309** | `test2` | 31,036 | `…PROMOTION.json` → `verification.test2_macro_f1` | MEASURED, ruling **OPEN** |
| VLM (adapted) | exact_match | **0.963** | frozen 1,000-question subset | 1,000 | `vlm/phase6_closure.json` → `why_usable_verified.adapted_test.exact_match` | MEASURED, **ACCEPTANCE-REJECTED** |
| VLM (adapted) | F1 | **0.96432** | same | 1,000 | `…phase6_closure.json` → `…adapted_test.f1` | MEASURED, **ACCEPTANCE-REJECTED** |
| Router | overall **ungated** accuracy | **0.965116** | val, corpus-limited | 86 | `router/threshold_sweep_val.json` → `overall_ungated_accuracy` | MEASURED — **TEST NOT RUN** |
| Calibration | ECE before / after | **0.013755 → 0.014929** | val, T = 0.9773 | 16,441 | `calibration_v001.json` → `metrics.ece_before` / `.ece_after` | MEASURED — **worse** |
| System | end-to-end accuracy | — | — | — | — | **NOT RUN — none exists** |

---

## 9. Calibration — a measured negative result

Temperature scaling is **enabled** in the frozen configuration (`confidence.temperature_scaling: true`,
`confidence.calibration_file: calibration_v001.json`) and applied by
`evidence.confidence.TemperatureCalibration` as `sigmoid(logit(z)/T)` for a scalar `z` and
`softmax(logits/T)` for a distribution. Source: `artifacts/calibration_v001.json`.

| Field | Value | | Field | Value |
|---|---|---|---|---|
| `method` | `temperature_scaling` | | `temperature` | **0.9772731820958189** |
| `fitted_on` / `n_samples` | Val / **16,441** | | `n_classes` / `space` | 19 / `multiclass_logits` |
| `objective` | `mean_negative_log_likelihood` | | `optimizer` | `golden_section_on_log_temperature` (200 iters, `hit_bound: false`) |
| NLL before → after | 0.689741 → 0.689631 (Δ 0.00011) | | **ECE before → after** | **0.013755 → 0.014929** |
| `ece_improvement` | **−0.001174** (negative ⇒ did **not** help) | | `n_bins` | 15 |
| scope | `change_vqa` only — "Other specialists emit their own raw scores and are unaffected." | | `type_mask_applied` | `false` |
| held-out splits excluded | `[Test, Test2]` | | | |

**The honest reading: the ECE got worse.** Temperature scaling reduced the NLL very slightly (0.00011)
but **increased** the expected calibration error from 0.013755 to 0.014929. It is retained **only
because it is part of the frozen configuration** — not because it helped. This is a measured negative
result and is reported as one (`docs/BENCHMARKS.md` §4.7, `docs/MODELS.md` §5).

**Two caveats on the number.** The artifact notes that "ECE is bin-count sensitive and is not an
aggregate score", and the `reliability_diagram` it carries is the **pre-scaling** curve (`ece 0.013755`),
labelled as such — the calibrated curve is **NOT plotted** (`docs/LIMITATIONS.md` §3.22).

---

## 10. Acceptance status

| Artifact | Metrics | Acceptance | Notes |
|---|---|---|---|
| `change` | VERIFIED | accepted (shipped) | the only `VERIFIED` headline |
| `grounding` | measured (2 protocols × 2 decode variants) | shipped | trained head is the production default; zero-shot is a labelled fallback (D-4) |
| `optical_sar` | measured | **ruling OPEN** | accuracy 0.931 always with macro-F1 0.434161 |
| `change_vqa` | measured (2 test sets) | **ruling OPEN** | test + test2 both reported |
| `router` | measured (val only) | shipped; **test NOT RUN** | 0.965116 is validation, ungated, n = 86 |
| **`vlm`** | usable (exact_match 0.963, F1 0.96432) | **ACCEPTANCE-REJECTED** | deployed path uses the unadapted model |

**`USABLE_VERIFIED` ≠ `ACCEPTANCE-ACCEPTED`.** The VLM adapter works and is not promoted. The two
questions — *is this the artifact we trained, and does it work?* versus *did it clear the bar we
predeclared before looking?* — are kept separate on purpose (`docs/PHASE6_CLOSURE.md` §1).

---

## 11. Evaluation gaps (stated, not hidden)

| Gap | State |
|---|---|
| **System-level end-to-end benchmark** | **NOT RUN — none exists.** No end-to-end accuracy is claimed. |
| **Router test split** | **NOT RUN** (`test_split_touched: false`) |
| **Benchmark adapters** | **NOT RUN** |
| **End-to-end latency benchmark** | **NOT RUN** (per-specialist latency recorded only incidentally) |
| **Cross-dataset generalisation** | **NOT RUN** — each specialist is evaluated only on its own training-family split |
| **Human evaluation** | **NOT RUN** |
| **Robustness / adversarial evaluation** | **NOT RUN** |
| **Statistical significance for most metrics** | only the grounding 448-vs-224 decision has a paired test with a CI; other per-task numbers are point estimates |
| **Calibrated reliability curve** | **NOT plotted** |
| **BigEarthNet label semantics** | the local subset is **100 % single-label** vs the official 1–11 multi-label scheme, so its metrics are **not comparable** to published numbers |

---

## 12. Limitations

A condensed catalogue; the full version is [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Per-artifact
limitations are in §7; evaluation gaps in §11.

**Model quality.** Grounding absolute IoU is low (0.2838 / 0.2566) and **protocol-sensitive**.
Optical-SAR accuracy is carried by common classes — **0.931 with macro-F1 0.434161**. Change-VQA is weak
on rare classes (0.697626/0.378373 and 0.651469/0.372309). The live optical-SAR service returns a **bare
class index** (`class_18`), not a CLC label. The VLM adapter is **not accepted**. Calibration made ECE
**worse**.

**Router.** 0.965116 is **validation, ungated, n = 86**, corpus-limited; the test split was **NOT RUN**.
Known residuals: *"What is the new runway?"* reads `change`; *"How much built-up area was added?"*
under-triggers `vqa`; with one asset the console *reads* `change` while dispatch correctly falls back to
`change_vqa` (intentional, but visually surprising).

**Evaluation.** No end-to-end benchmark; no cross-dataset, human or robustness evaluation; most metrics
are point estimates without confidence intervals. The BigEarthNet local subset is **100 % single-label**,
so its metrics are **not comparable** to published multi-label numbers.

**Operational.** Transient tunnel gaps (B-07) — a request can hang or return 504; patch prepared but
**NOT deployed**. `OPEN`. `/api/health` `codespace_name` carries a trailing newline (B-02) — cosmetic.
`OPEN`. Cold start is tens of seconds; single-region, no HA; no database, auth or queue (stateless by
design).

**Packaging and licensing.** **No `LICENSE` file exists** in the source repository. `OPEN`. The six
artifacts require their pinned backbones, which are **not** redistributed.

**Documentation.** `docs/FINAL_DELIVERY_REPORT.md` §6 is stale (it lists the bundled EO change pair as
DEGRADED and B-01 as BLOCKED; both were resolved on 2026-09-25). The original master plan describes a
**superseded deployment** (Gradio GUI + HF Space + ZeroGPU + Railway); the shipped system is a static
frontend + Render + Codespace tunnel serving JSON.

**Explicit non-claims.** No state-of-the-art claim; no production-readiness claim for model quality; no
claim that the trained heads generalise beyond their training-family splits; no claim that calibration
improves confidence; no claim that the VLM adapter is accepted; no end-to-end accuracy claim; no
robustness claim; no geolocation-accuracy claim; not a safety-, legal- or life-critical tool.

---

## 13. Training summary

All six artifacts are small modules on frozen backbones, trained with **seed 42** and recording the
frozen config hash **`78f1e3700da15aa1`**.

| Artifact | Where it trained | Precision | Notable settings |
|---|---|---|---|
| `change` | GPU (eval artifact: `device cuda`, torch 2.10.0+cu128) | — | STANet-style; ResNet-18 pretrained; PAM; bce 0.5 + dice 0.5 |
| `change_vqa` | **external GPU (Kaggle)** | fp16 | epoch 8 selected on val answer accuracy 0.700018; early stopping |
| `optical_sar` | in-project | — | arm A retained (A 0.837100 vs B 0.839100, floor 0.0285); 10 runs × 2 arms × 5 seeds |
| `grounding` | CPU | — | 20 epochs; grid 7×7; resolution frozen at 224 by pre-registered test |
| `router` | **CPU** | — | frozen encoder, cached embeddings; 20 epochs / 4,096 vectors in 0.28 s |
| `vlm` | **external GPU (Kaggle, T4)** | fp16 | LoRA r=16 α=32; 224 text-projection modules; vision tower untouched |

**Precision.** `training.precision: fp16` because the target GPU (T4) is compute capability 7.5 — bf16 is
unavailable there (finding C-6). The loader validates the value is one of `fp16|bf16|fp32`.

**Provenance discipline.** `change_vqa` was promoted from a Kaggle export with 93/0/0 verification
checks and a byte-identical source copy. The VLM adapter's Phase 6 closure was reached **without
retraining or modifying** the adapter — the promoted weights are the end-of-training top-level save, and
the closure record is generated from the evidence rather than restated. Full procedures:
[`docs/TRAINING.md`](docs/TRAINING.md); dataset provenance: [`docs/DATASETS.md`](docs/DATASETS.md).

---

## 14. Provenance and verification

| Item | Location |
|---|---|
| Byte-exact manifest (generated from disk) | [`models/manifest.json`](models/manifest.json) |
| Checksums | [`models/checksums.sha256`](models/checksums.sha256) |
| Metric verification tool / output | [`tools/verify_readme_metrics.py`](tools/verify_readme_metrics.py) · [`tools/readme_metrics_report.txt`](tools/readme_metrics_report.txt) |
| HF release verification (re-downloaded, 6/6 MATCH) | [`HF_RELEASE_VERIFICATION.md`](HF_RELEASE_VERIFICATION.md) |
| Release manifest (every file, size + sha256) | [`RELEASE_MANIFEST.md`](RELEASE_MANIFEST.md) |
| Full documentation / repository front page | [`docs/`](docs/) · [`README.md`](README.md) |

**Verification chain.** (1) `models/manifest.json` and `models/checksums.sha256` are **generated by
reading the files** (`release/tools/generate_model_manifest.py`); no byte count or hash is typed by hand.
(2) `tools/verify_readme_metrics.py` walks every quoted metric to its source artifact; result
**`ALL CLAIMS VERIFIED`** (20/20), with status assertions (VLM `ACCEPTANCE-REJECTED`, router
`corpus_limited` n = 86, calibration `ece_improvement` negative) confirmed. (3)
`release/tools/hf_verify.py` re-downloads each artifact over direct HTTPS and hashes the received bytes;
**6/6 MATCH, 0 failed**. (4) Two artifact digests agree with values recorded independently at
promotion/fit time (§3).

**Hugging Face release.** `thundercode/SatQuery` (public), HEAD `bf2779e18fcaa7476b93a48a978f08c108dfdfb7`,
`lastModified 2026-09-25T21:46:52Z`, 42 files on the Hub. No secret was uploaded; the token used is not
written into any released file (`HF_RELEASE_VERIFICATION.md` §8).

**GitHub release.** The curated public repository target is `Anish-lab-blip/SatQuery-AI`; the release
tree is staged and its links verified. At the time of `release/RELEASE_EXECUTION_CHECKLIST.md` Phase 5,
the push was **BLOCKED** because the fine-grained token was read-only for repository contents
(`403 Resource not accessible by personal access token`) — an owner action, not a defect in this release.
Where this card and a live repository disagree, treat the live repository as authoritative for
publication state and this card as authoritative for artifact identity.

---

## 15. Licence

The project ships **no licence file**; a licence must be selected by the owner before public release of
the *code* (`docs/LIMITATIONS.md` §5, `RELEASE_MANIFEST.md`). This is an **OPEN** item. The Hugging Face
card declares `license: other` because the correct licence has not yet been chosen.

**Model weights carry the terms of their backbone licences.** The six artifacts are small modules, but
they depend on and are intended to be used with: `sentence-transformers/all-MiniLM-L6-v2`;
`HuggingFaceTB/SmolVLM-500M-Instruct`; `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`);
`antofuller/CROMA` (`CROMA_base.pt`); and torchvision ResNet-18 (`IMAGENET1K_V1`). **Backbones are not
redistributed here.** Consult each backbone's Hugging Face page for the authoritative licence — the
licence labels above are recorded for convenience and should be verified at the source before reuse.

---

## 16. Citation

If you use this work, cite the project repository:

```bibtex
@misc{satquery_ai_2026,
  title  = {SatQuery AI: A Modular Router-and-Specialists System for Satellite Imagery Question Answering},
  author = {SatQuery AI},
  year   = {2026},
  note   = {Public release: https://github.com/Anish-lab-blip/SatQuery-AI}
}
```

**Model card version.** This card documents release **1.0.0** (2026-09-25), frozen config hash
`78f1e3700da15aa1`. The changelog is [`docs/CHANGELOG.md`](docs/CHANGELOG.md).