docs/MODELS.md

# Models

**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `ATTEMPTED` · `NOT RUN` · `BLOCKED` ·
`DEFERRED` · `REJECTED`.

SatQuery AI trains **six** artifacts. Four are task heads, one is a router adapter, one is a LoRA
adapter. Every backbone is **frozen** and publicly pinned by revision in `configs/base.yaml` — the
project trains small modules on top of frozen encoders, not end-to-end networks.

> **This file is the human-readable companion to the machine-generated
> [`../models/manifest.json`](../models/manifest.json) and
> [`../models/checksums.sha256`](../models/checksums.sha256) (Phase 3).** Where the two disagree,
> the generated manifest wins — it is computed from the files, this document is written by hand.

---

## 1. The six trained artifacts

| # | Task | Artifact path | Bytes | Kind | Backbone (frozen) |
|---|---|---|---|---|---|
| 1 | `change` | `artifacts/change/levir_change_v001/head.pt` | 63,231,009 | trained head | STANet-style, ResNet-18 + PAM |
| 2 | `change_vqa` | `artifacts/change_vqa/run/head.pt` | 5,822,809 | trained head | over the change encoder's features |
| 3 | `optical_sar` | `artifacts/optical_sar/fusion_head_production_v001/head.pt` | 14,427,457 | trained head (production) | CROMA-base (frozen), 19-class head |
| 4 | `grounding` | `artifacts/grounding/remoteclip_grounding_v001/head.pt` | 12,639,041 | trained head | RemoteCLIP ViT-B/32 (frozen) |
| 5 | `router` | `artifacts/router/router_adapter_v001/adapter.pt` | 211,961 | trained adapter | `all-MiniLM-L6-v2` (frozen) |
| 6 | `vlm` | `.scratch/phase6_real_adapter/phase6_adapter/adapter_model.safetensors` | 34,798,048 | **LoRA adapter** (PEFT) | `HuggingFaceTB/SmolVLM-500M-Instruct` (frozen) |

Training checkpoints also exist (`checkpoint_last.pt`, `checkpoint-1500`, `checkpoint-2000`) and are
**not** the released artifacts — they are archived as provenance.

## 2. Backbones — pinned, frozen, never retrained

| Role | Repository | Revision | Notes |
|---|---|---|---|
| Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | 90.9 MB, 22,713,216 params, 384-dim embeddings, tokenizer ceiling 256; truncation set to 128 |
| VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | ~1015 MB safetensors |
| Grounding | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`) | `bf1d8a3ccf2d` | 605.2 MB; transformer width 768, **projected** dim 512 |
| Optical-SAR | `antofuller/CROMA` (`CROMA_base.pt`) | `0dd28e3d633b` | 777.6 MB; `encoder_dim` 768; `image_resolution` 120 |

These are resolved from the Hugging Face Hub on first use. **No backbone weights are redistributed**
by this project's release — see §6.

## 3. Per-artifact detail

### 3.1 Change (`change`) — `IMPLEMENTED`, `VERIFIED`

STANet-style Siamese change detector. Encoder ResNet-18, self-attention mode **PAM**, tile size 256,
threshold 0.50, minimum component 32 px. Loss is BCE (0.5) + Dice (0.5). Trained on LEVIR-CD-256
(train 7120 / val 1024 / test 2048).

**Measured** on the LEVIR-CD-256 test split (n = 2048, threshold 0.50):

| Metric | Value |
|---|---|
| pooled IoU | **0.8122** |
| macro IoU | **0.8457** |
| pooled F1 | **0.8964** |

This is the only task whose headline number carries the `VERIFIED` tag, because it is the only one
measured against a single, immutable public test split with a frozen threshold.

### 3.2 Change-VQA (`change_vqa`) — `IMPLEMENTED`, `MEASURED`, ruling **OPEN**

A head that answers natural-language change questions over a temporal pair. It is the dispatch
target for change-style questions when only one asset is attached (see
[`ARCHITECTURE.md`](ARCHITECTURE.md) §4).

**Measured on two test sets — both are reported; quoting only the better one would be selective:**

| Test set | accuracy | macro F1 |
|---|---|---|
| `test` (n = 39,686) | **0.697626** | **0.378373** |
| `test2` | **0.651469** | **0.372309** |

The wide gap between accuracy and macro-F1 means the head is carried by common classes and performs
poorly on rare ones. The ruling is **OPEN** — no promotion/acceptance decision has been recorded.

### 3.3 Optical-SAR fusion (`optical_sar`) — `IMPLEMENTED`, `MEASURED`, ruling **OPEN**

Uses frozen CROMA-base to produce optical (768), SAR (768) and joint (768) embeddings, concatenates
them with the 12 optical and 2 SAR channel descriptors, and feeds a 19-class head:

```
input_dim = 3 * 768 + 12 + 2 = 2318   →   hidden 512   →   num_classes 19   (BigEarthNet CLC)
```

The **availability mask is consumed by the fusion head, not by CROMA** — CROMA always sees the
canonical channel counts (12 optical, 2 SAR).

**Measured** (production head `fusion_head_production_v001`, pre-registered 115-class protocol,
held-out test n = 4000):

| Metric | Value |
|---|---|
| accuracy | **0.931** |
| macro F1 | **0.434161** |

**Both numbers must travel together.** The high accuracy with a low macro-F1 reflects class
imbalance across 19 classes. The pre-registered metric JSON records `macro_f1_denominator` and
`classes_present`/`classes_absent` so the denominator is auditable. The ruling is **OPEN**.

**Limitation:** the live service returns a bare class index (`class_18`), not a human-readable label.

### 3.4 Grounding (`grounding`) — `IMPLEMENTED`, `MEASURED` — **two protocols**

A trainable head over the frozen RemoteCLIP ViT-B/32 encoder. Per-cell feature is
`concat([patch, text, patch·text, global_pool])` = `4 × 512 = 2048` (enforced at config load). Cells
are assigned by ground-truth box centre (`cell_relative` decode). Objectness BCE is weighted **20×**
because only ~1 of 49 cells is positive; unweighted, the optimum collapses to "no object".

Image resolution is frozen at **224** — 448 was evaluated and **rejected** (see §4).

**Measured on VRSBench (n = 16,159), reported under two protocols:**

| Protocol | mean best IoU | recall@0.5 |
|---|---|---|
| canonical (head threshold decode) | **0.2838** | **0.2198** |
| matched6 | **0.2566** | **0.1938** |

Two further decode variants exist and are reported for completeness — a reviewer must be able to see
the whole grid, not one cell of it:

| Variant | mean best IoU |
|---|---|
| head argmax decode (canonical) | **0.1215** |
| zero-shot matched (no trained head) | **0.0972** |

The trained head beats the zero-shot baseline by a wide margin, which is the point of the head; the
absolute IoU is low, which is an honest limitation.

### 3.5 Router (`router`) — `IMPLEMENTED`, `MEASURED`, **TEST NOT RUN**

A **50,822-parameter adapter** over the frozen MiniLM encoder. Because the encoder is frozen,
embeddings are cached and the adapter trains on cached vectors — no GPU required (measured: 20
epochs / 4,096 vectors in 0.28 s on CPU). Splits are by **group** (template / hard-negative family),
never by example; hard-negative families are placed in the test split so their accuracy measures
generalisation, not memorisation.

**Measured:** overall **ungated** task accuracy **0.965116** on the validation split, **n = 86**,
corpus-limited. This number is (a) validation-only, (b) ungated, and (c) small. The router **test**
split was **NOT RUN**. Do not read 0.965116 as a test result.

### 3.6 VLM LoRA (`vlm`) — `IMPLEMENTED`, `MEASURED`, **ACCEPTANCE-REJECTED**

A PEFT LoRA adapter on frozen SmolVLM-500M-Instruct: `peft_type=LORA`, `r=16`, `alpha=32`,
`dropout=0.05`, targeting `model.text_model.*.{q,k,v,o,gate,up,down}_proj`. PEFT 0.19.1.

**Measured** on a frozen 1,000-question subset: exact_match **0.963**, F1 **0.96432** (+49.5 pp over
the unadapted baseline).

**Yet the artifact's status is `CLOSED` with headline `ACCEPTANCE-REJECTED`.** This is not a
contradiction — it is the project's central truthfulness distinction:

- **`USABLE_VERIFIED`** — the adapter demonstrably works (the metrics are real and reproducible).
- **`ACCEPTANCE-REJECTED`** — the adapter is *not accepted* for production promotion, on grounds
  recorded in `artifacts/vlm/phase6_closure.json` (`why_acceptance_rejected`).

The deployed caption/VQA path therefore uses the **unadapted** SmolVLM. Deployment success and model
acceptance are different claims, and this document keeps them apart.

## 4. Rejected and deferred model decisions

| Decision | Outcome | Evidence |
|---|---|---|
| Grounding image resolution 448 vs 224 | **224 chosen; 448 REJECTED** | 448 lost on every axis: mean best IoU −0.0147, recall@0.5 −0.0022, all recall thresholds lower, at 1.59× latency. Paired test: mean diff −0.0147, 95 % CI [−0.0160, −0.0134], t = −22.63; 448 better on 8.5 % of records, worse on 20.9 %. Pre-registered rule and the paired test **agree**. |
| VLM adapter promotion | **REJECTED** | metrics usable, acceptance rejected (§3.6) |
| Calibration | **kept but ineffective** | see §5 |
| optical-SAR / change-VQA rulings | **OPEN** | no decision recorded |

## 5. Calibration — `MEASURED`, **not an improvement**

Temperature scaling is enabled (`confidence.temperature_scaling: true`) with
`calibration_v001.json`. Fitted temperature **T = 0.9772732** on the validation split (n = 16,441).

| Metric | Before | After |
|---|---|---|
| ECE | 0.013755 | **0.014929** |
| NLL | 0.689741 | 0.689631 |

**ECE got worse** (`ece_improvement = −0.001174`). The scaling is retained because it is part of the
frozen configuration, **not** because it helped. The reliability diagram on the Benchmark page is
explicitly labelled **pre-scaling** so a reader cannot mistake it for the calibrated result. This is
recorded as a negative result, not smoothed over.

## 6. Distribution and licensing

- **Backbones are not redistributed.** They are fetched from the Hugging Face Hub at run time, pinned
  by revision. Their licences are their own (see each model's HF page).
- **The six trained artifacts are published by this project** on the Hugging Face Hub under
  `thundercode/SatQuery`, labelled by kind, each with its backbone dependency documented and each
  accompanied by a checksum. See [`../HF_RELEASE_VERIFICATION.md`](../HF_RELEASE_VERIFICATION.md).
- **No licence file exists in the source repository.** This is an **OPEN** item flagged in
  [`LIMITATIONS.md`](LIMITATIONS.md); the repository README instructs the owner to select one before
  any public release of *code*. Model weights carry the terms of their backbone licences.