Image-to-Text
PyTorch
Safetensors
PEFT
English
remote-sensing
satellite-imagery
earth-observation
change-detection
visual-grounding
image-captioning
visual-question-answering
optical-sar-fusion
sar
multimodal
lora
Instructions to use thundercode/SatQuery with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use thundercode/SatQuery with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
release: add MODEL_CARD.md
Browse files- MODEL_CARD.md +816 -86
MODEL_CARD.md
CHANGED
|
@@ -20,7 +20,7 @@ config_hash: 78f1e3700da15aa1
|
|
| 20 |
# Model Card β SatQuery AI
|
| 21 |
|
| 22 |
SatQuery AI answers natural-language questions about satellite imagery using a **router + specialists**
|
| 23 |
-
design. This card
|
| 24 |
explicit about what is measured, what is not, and what was rejected.
|
| 25 |
|
| 26 |
> **The six trained artifacts are small modules on top of frozen, publicly-pinned backbones. No
|
|
@@ -29,128 +29,855 @@ explicit about what is measured, what is not, and what was rejected.
|
|
| 29 |
|
| 30 |
Machine-readable identities (byte counts and sha256) are in
|
| 31 |
[`models/manifest.json`](models/manifest.json) and [`models/checksums.sha256`](models/checksums.sha256),
|
| 32 |
-
**generated by reading the files**. Where this card and the
|
| 33 |
-
wins.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
---
|
| 36 |
|
| 37 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
-
|
| 49 |
-
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
-
|
|
|
|
| 52 |
|
| 53 |
-
|
|
|
|
|
|
|
| 54 |
|---|---|---|
|
| 55 |
-
|
|
| 56 |
-
|
|
| 57 |
-
|
|
| 58 |
-
|
|
| 59 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
-
|
|
|
|
|
|
|
| 62 |
|
| 63 |
-
|
| 64 |
-
-
|
| 65 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
|
| 67 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
|
| 69 |
- **Safety-, legal- or life-critical decisions.** No accuracy, calibration or robustness guarantee is
|
| 70 |
-
offered
|
|
|
|
| 71 |
- **Operational geospatial production** without independent validation.
|
| 72 |
-
- **Any use of the VLM adapter as a production model** β it is
|
| 73 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
-
##
|
| 76 |
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
|
| 87 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
|
| 95 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
|
| 97 |
-
|
|
|
|
|
|
|
| 98 |
|---|---|---|
|
| 99 |
-
|
|
| 100 |
-
|
|
| 101 |
-
|
|
| 102 |
-
|
|
| 103 |
-
|
|
| 104 |
-
| **
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
-
|
| 107 |
-
|
|
|
|
| 108 |
|
| 109 |
-
|
|
|
|
|
|
|
|
|
|
| 110 |
|
| 111 |
-
|
| 112 |
-
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
-
|
|
|
|
|
|
|
|
|
|
| 118 |
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
its metrics are **not comparable** to published numbers.
|
| 123 |
-
- The optical-SAR service returns a bare class index, not a label.
|
| 124 |
-
- Known router residuals exist (e.g. *"What is the new runway?"* reads `change`).
|
| 125 |
-
- No `LICENSE` file exists in the source repository.
|
| 126 |
|
| 127 |
-
|
| 128 |
|
| 129 |
-
##
|
| 130 |
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 135 |
|
| 136 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
|
| 138 |
| Item | Location |
|
| 139 |
|---|---|
|
| 140 |
-
| Byte-exact manifest | `models/manifest.json` |
|
| 141 |
-
| Checksums | `models/checksums.sha256` |
|
| 142 |
-
| Metric verification tool | `tools/verify_readme_metrics.py` |
|
| 143 |
-
|
|
| 144 |
-
|
|
| 145 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
|
| 147 |
-
##
|
| 148 |
|
| 149 |
-
The project ships **no licence file**; a licence must be selected by the owner before public release
|
| 150 |
-
|
| 151 |
-
|
| 152 |
|
| 153 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
|
| 155 |
If you use this work, cite the project repository:
|
| 156 |
|
|
@@ -162,3 +889,6 @@ If you use this work, cite the project repository:
|
|
| 162 |
note = {Public release: https://github.com/Anish-lab-blip/SatQuery-AI}
|
| 163 |
}
|
| 164 |
```
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
# Model Card β SatQuery AI
|
| 21 |
|
| 22 |
SatQuery AI answers natural-language questions about satellite imagery using a **router + specialists**
|
| 23 |
+
design. This card documents the **six trained artifacts** released by the project. It is deliberately
|
| 24 |
explicit about what is measured, what is not, and what was rejected.
|
| 25 |
|
| 26 |
> **The six trained artifacts are small modules on top of frozen, publicly-pinned backbones. No
|
|
|
|
| 29 |
|
| 30 |
Machine-readable identities (byte counts and sha256) are in
|
| 31 |
[`models/manifest.json`](models/manifest.json) and [`models/checksums.sha256`](models/checksums.sha256),
|
| 32 |
+
**generated by reading the files** (`release/tools/generate_model_manifest.py`). Where this card and the
|
| 33 |
+
generated manifest disagree, the manifest wins β it is computed from disk, this card is written by hand.
|
| 34 |
+
|
| 35 |
+
**Companion documents** (same depth, same no-fabrication rule): [`docs/MODELS.md`](docs/MODELS.md) Β·
|
| 36 |
+
[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) Β· [`docs/TRAINING.md`](docs/TRAINING.md) Β·
|
| 37 |
+
[`docs/EVALUATION.md`](docs/EVALUATION.md) Β· [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md) Β·
|
| 38 |
+
[`docs/RESEARCH_NOTES.md`](docs/RESEARCH_NOTES.md) Β·
|
| 39 |
+
[`docs/architecture/05-specialists.md`](docs/architecture/05-specialists.md).
|
| 40 |
|
| 41 |
---
|
| 42 |
|
| 43 |
+
## Table of contents
|
| 44 |
+
|
| 45 |
+
1. [How to read this card](#1-how-to-read-this-card)
|
| 46 |
+
2. [Overview](#2-overview)
|
| 47 |
+
3. [The six artifacts in this release](#3-the-six-artifacts-in-this-release)
|
| 48 |
+
4. [Backbone dependencies β frozen, pinned by revision](#4-backbone-dependencies--frozen-pinned-by-revision)
|
| 49 |
+
5. [Intended use](#5-intended-use)
|
| 50 |
+
6. [Out-of-scope use](#6-out-of-scope-use)
|
| 51 |
+
7. [Per-artifact reference](#7-per-artifact-reference)
|
| 52 |
+
8. [Full measured-performance table](#8-full-measured-performance-table)
|
| 53 |
+
9. [Calibration β a measured negative result](#9-calibration--a-measured-negative-result)
|
| 54 |
+
10. [Acceptance status](#10-acceptance-status)
|
| 55 |
+
11. [Evaluation gaps](#11-evaluation-gaps)
|
| 56 |
+
12. [Limitations](#12-limitations)
|
| 57 |
+
13. [Training summary](#13-training-summary)
|
| 58 |
+
14. [Provenance and verification](#14-provenance-and-verification)
|
| 59 |
+
15. [Licence](#15-licence)
|
| 60 |
+
16. [Citation](#16-citation)
|
| 61 |
|
| 62 |
+
---
|
| 63 |
+
|
| 64 |
+
## 1. How to read this card
|
| 65 |
+
|
| 66 |
+
**The single most important rule in this document: do not fabricate.** Every byte count, sha256,
|
| 67 |
+
hyperparameter and metric below comes from a file that was read, and each one names its source. Where a
|
| 68 |
+
fact is not established, this card writes `UNKNOWN β not established from the available evidence`
|
| 69 |
+
rather than estimating.
|
| 70 |
+
|
| 71 |
+
**Status vocabulary.** Every substantive claim carries one of: `IMPLEMENTED` Β· `VERIFIED` Β· `MEASURED` Β·
|
| 72 |
+
`ATTEMPTED` Β· `NOT RUN` Β· `BLOCKED` Β· `DEFERRED` Β· `REJECTED` Β· `OPEN` Β· `RESOLVED` Β· `CLOSED`.
|
| 73 |
+
|
| 74 |
+
**The facts most easily stated wrongly, and therefore stated repeatedly:**
|
| 75 |
+
|
| 76 |
+
| Fact | Correct statement |
|
| 77 |
+
|---|---|
|
| 78 |
+
| Grounding | measured under **two protocols** (canonical 0.2838 / 0.2198; matched6 0.2566 / 0.1938) **and two decode variants** (head_argmax 0.1215; zero-shot 0.0972). Never quote one alone. |
|
| 79 |
+
| Calibration | ECE went **0.013755 β 0.014929 β worse**. Retained only because it is in the frozen config. |
|
| 80 |
+
| VLM adapter | metrics **usable** (exact_match 0.963) but status **ACCEPTANCE-REJECTED**. USABLE β ACCEPTED. |
|
| 81 |
+
| Optical-SAR | accuracy **0.931** with macro-F1 **0.434161**; ruling **OPEN**. Never accuracy without macro-F1. |
|
| 82 |
+
| Change-VQA | **two** test sets: test 0.697626/0.378373 and test2 0.651469/0.372309; ruling **OPEN**. |
|
| 83 |
+
| Router | **0.965116 is validation, ungated, n = 86**; the **test split was NOT RUN**. |
|
| 84 |
+
| End-to-end benchmark | **does not exist**; no system-level accuracy is claimed. |
|
| 85 |
+
| Change | pooled IoU 0.8122 / macro IoU 0.8457 / pooled F1 0.8964 β the **only** `VERIFIED` headline. |
|
| 86 |
+
|
| 87 |
+
---
|
| 88 |
+
|
| 89 |
+
## 2. Overview
|
| 90 |
|
| 91 |
+
SatQuery AI is a **router-and-specialists** system: a frozen sentence encoder plus a small trained
|
| 92 |
+
adapter classify a query into one of six tasks; a deterministic planner dispatches it to the
|
| 93 |
+
appropriate specialist; each specialist returns a structured `ResultEnvelope` carrying evidence and a
|
| 94 |
+
confidence value. The design is **CPU-first** and **frozen-backbone** β small modules are trained on top
|
| 95 |
+
of pretrained encoders, and no encoder is fine-tuned end to end.
|
| 96 |
|
| 97 |
+
**The six tasks** (`configs/base.yaml` β `router.tasks`):
|
| 98 |
+
`vqa` Β· `caption` Β· `grounding` Β· `change` Β· `optical_sar` Β· `unsupported`.
|
| 99 |
|
| 100 |
+
**The six trained artifacts** (details in Β§3):
|
| 101 |
+
|
| 102 |
+
| Task | What it is | Size |
|
| 103 |
|---|---|---|
|
| 104 |
+
| `change` | STANet-style Siamese change detector (ResNet-18 + PAM) | 63,231,009 B |
|
| 105 |
+
| `change_vqa` | two-stage change-reasoning head (`change_vqa_head_v1`) | 5,822,809 B |
|
| 106 |
+
| `optical_sar` | CROMA-base fusion head (2318 β 512 β 19) | 14,427,457 B |
|
| 107 |
+
| `grounding` | RemoteCLIP grounding head (feature 2048 β hidden 512) | 12,639,041 B |
|
| 108 |
+
| `router` | five-head intent adapter over frozen MiniLM | 211,961 B |
|
| 109 |
+
| `vlm` | PEFT LoRA adapter on SmolVLM-500M text projections | 34,798,048 B |
|
| 110 |
+
|
| 111 |
+
Total released weight payload: **131,130,325 bytes (~125 MiB)** (`HF_RELEASE_VERIFICATION.md` Β§4).
|
| 112 |
+
|
| 113 |
+
**What is not trained here.** MiniLM, SmolVLM-500M, RemoteCLIP ViT-B/32 and CROMA-base are frozen and
|
| 114 |
+
not redistributed. The one nuance is the change detector's ResNet-18, which is *loaded pretrained*
|
| 115 |
+
(`pretrained_used: true`) and *trained in-project* as part of the change head β so its weights are part
|
| 116 |
+
of the released `change/head.pt`, not a separately-distributed backbone
|
| 117 |
+
(`artifacts/change/eval_test/eval_result.json` β `checkpoint_embedded_config`).
|
| 118 |
+
|
| 119 |
+
**No end-to-end accuracy is claimed anywhere.** The router β specialist β envelope pipeline has never
|
| 120 |
+
been scored end to end. What exists is per-specialist metrics on their own training-family splits (Β§8)
|
| 121 |
+
and a behavioural live-validation record that proves the pipeline *runs and routes* β 3 passes Γ 8
|
| 122 |
+
cases, 8/8 each, **24 live runs, 0 mock nodes**, trace fill **94.4444 %** (`docs/BENCHMARKS.md` Β§5,
|
| 123 |
+
`docs/RESEARCH_NOTES.md` Β§3.3).
|
| 124 |
+
|
| 125 |
+
---
|
| 126 |
+
|
| 127 |
+
## 3. The six artifacts in this release
|
| 128 |
|
| 129 |
+
Reproduced from [`models/manifest.json`](models/manifest.json) β `artifacts[*]`, cross-checked against
|
| 130 |
+
[`models/checksums.sha256`](models/checksums.sha256). Every artifact carries `status: "PRESENT"` and
|
| 131 |
+
`config_hash: "78f1e3700da15aa1"`.
|
| 132 |
|
| 133 |
+
| # | `id` | Task | Kind | File (HF path) | Bytes | sha256 (full) |
|
| 134 |
+
|---|---|---|---|---|---|---|
|
| 135 |
+
| 1 | `change_head` | `change` | trained head | `change/head.pt` | 63,231,009 | `c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa` |
|
| 136 |
+
| 2 | `change_vqa_head` | `change_vqa` | trained head | `change_vqa/head.pt` | 5,822,809 | `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` |
|
| 137 |
+
| 3 | `optical_sar_fusion_head` | `optical_sar` | trained head | `optical_sar/head.pt` | 14,427,457 | `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab` |
|
| 138 |
+
| 4 | `grounding_head` | `grounding` | trained head | `grounding/head.pt` | 12,639,041 | `93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb` |
|
| 139 |
+
| 5 | `router_adapter` | `router` | trained adapter | `router/adapter.pt` | 211,961 | `8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` |
|
| 140 |
+
| 6 | `vlm_lora_adapter` | `vlm` | LoRA adapter | `vlm/adapter_model.safetensors` | 34,798,048 | `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` |
|
| 141 |
|
| 142 |
+
The manifest also records each artifact's **original repository path** and its **source metric
|
| 143 |
+
artifact**:
|
| 144 |
+
|
| 145 |
+
| # | `id` | `path` (source repo) | `source_metric_artifact` |
|
| 146 |
+
|---|---|---|---|
|
| 147 |
+
| 1 | `change_head` | `artifacts/change/levir_change_v001/head.pt` | `artifacts/change/eval_test/eval_result.json` |
|
| 148 |
+
| 2 | `change_vqa_head` | `artifacts/change_vqa/run/head.pt` | `artifacts/change_vqa/run/PROMOTION.json` |
|
| 149 |
+
| 3 | `optical_sar_fusion_head` | `artifacts/optical_sar/fusion_head_production_v001/head.pt` | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` |
|
| 150 |
+
| 4 | `grounding_head` | `artifacts/grounding/remoteclip_grounding_v001/head.pt` | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json` |
|
| 151 |
+
| 5 | `router_adapter` | `artifacts/router/router_adapter_v001/adapter.pt` | `artifacts/router/threshold_sweep_val.json` |
|
| 152 |
+
| 6 | `vlm_lora_adapter` | `.scratch/phase6_real_adapter/phase6_adapter/adapter_model.safetensors` | `artifacts/vlm/phase6_closure.json` |
|
| 153 |
+
|
| 154 |
+
**`kind` semantics.** `trained_head` = a module trained in-project on a frozen encoder, loaded via
|
| 155 |
+
`torch.load` of a `state_dict` (or the module's own loader). `trained_adapter` = a small classifier over
|
| 156 |
+
a frozen sentence encoder's cached embeddings, loaded via `IntentAdapter.from_config_dict` +
|
| 157 |
+
`load_state_dict` (`router/adapter.py`). `lora_adapter` = a PEFT LoRA delta attached at load time via
|
| 158 |
+
`peft.PeftModel.from_pretrained(model, dir)` (`specialists/vqa/model.py`).
|
| 159 |
+
|
| 160 |
+
**Two independent cross-checks (not self-consistency).** The manifest is generated by hashing the files
|
| 161 |
+
on disk; for two artifacts the computed digest can be compared against a value recorded independently,
|
| 162 |
+
at a different time, by a different process:
|
| 163 |
+
|
| 164 |
+
- **`change_vqa` = `cfae5e43β¦d63a82a`.** Equals `artifacts/change_vqa/run/PROMOTION.json` β
|
| 165 |
+
`artifact.sha256`, the digest recorded in the Kaggle run record *before* promotion
|
| 166 |
+
(`source.checkpoint_sha256_in_run_record`), and `artifacts/calibration_v001.json` β
|
| 167 |
+
`provenance.checkpoint_sha256` (recorded when the temperature was fitted β a separate step).
|
| 168 |
+
`PROMOTION.json` β `source.hash_agrees_across` records the digest agreeing across
|
| 169 |
+
`model_metadata.json`, `run_record.json` and `hashes.json`, with `byte_identical_to_source: true` and
|
| 170 |
+
`artifact.weights_modified: false`.
|
| 171 |
+
- **`vlm` = `07c76a75β¦a5adf5e`.** Equals `artifacts/vlm/phase6_closure.json` β
|
| 172 |
+
`why_usable_verified.adapter_provenance` β `adapter_verification.json` β `weights_file_sha256`, and the
|
| 173 |
+
adapter's own `ARTIFACT_SHA256SUMS.json`, against which the 14-file directory was verified
|
| 174 |
+
(`manifest_check.clean: true`, 14/14 present).
|
| 175 |
+
|
| 176 |
+
A third, independent re-download check: `release/tools/hf_verify.py` re-downloads each artifact over
|
| 177 |
+
direct HTTPS and hashes the received bytes β **6/6 MATCH, 0 failed**
|
| 178 |
+
([`HF_RELEASE_VERIFICATION.md`](HF_RELEASE_VERIFICATION.md) Β§5).
|
| 179 |
+
|
| 180 |
+
**On `parameters: null`.** Four of six artifacts record `parameters: null` deliberately β the generator
|
| 181 |
+
does not open checkpoints (that would make generation depend on the model code and torch). Counts
|
| 182 |
+
measured elsewhere appear in Β§7 with their source; where a count is not established this card writes
|
| 183 |
+
`UNKNOWN β not established from the available evidence`. Training checkpoints are **not** released
|
| 184 |
+
artifacts: the VLM adapter's `checkpoint-1500/`/`checkpoint-2000/` are provenance only, and the promoted
|
| 185 |
+
adapter is the top-level end-of-training save, not `checkpoint-2000` (Β§7.6).
|
| 186 |
+
|
| 187 |
+
---
|
| 188 |
+
|
| 189 |
+
## 4. Backbone dependencies β frozen, pinned by revision
|
| 190 |
+
|
| 191 |
+
Backbones are resolved from the Hugging Face Hub on first use, **pinned by revision** β a moving `main`
|
| 192 |
+
would make every benchmark number unreproducible.
|
| 193 |
+
|
| 194 |
+
| Role | Repository | Revision | Size | Measured identity | Notes |
|
| 195 |
+
|---|---|---|---|---|---|
|
| 196 |
+
| Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | 90.9 MB | 22,713,216 params, 384-dim | tokenizer ceiling **256**; truncation **128** |
|
| 197 |
+
| VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | ~1015 MB safetensors | 516,165,824 params (base) | processor `longest_edge` must be pinned (F5-2) |
|
| 198 |
+
| Grounding | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`) | `bf1d8a3ccf2d` | 605.2 MB | 151,277,313 params; width **768**, projected **512** | patch 32; 7Γ7 tokens at 224 |
|
| 199 |
+
| Optical-SAR | `antofuller/CROMA` (`CROMA_base.pt`) | `0dd28e3d633b` | 777.6 MB (777,563,846 B) | 194,365,440 params; `encoder_dim` 768 | resolution 120; asymmetric `s1_depth=6`, `s2_depth=12` |
|
| 200 |
+
| Change encoder | β (torchvision) | β | β | ResNet-18, `IMAGENET1K_V1` | `pretrained_used: true` in the artifact |
|
| 201 |
+
|
| 202 |
+
All pins are declared in `configs/base.yaml` under the `router:`, `vlm:`, `grounding:` and `croma:`
|
| 203 |
+
blocks and are validated at load time by `core/config.py`. Backbone licence terms are each repository's
|
| 204 |
+
own β see Β§15. **Backbones are not redistributed here.**
|
| 205 |
+
|
| 206 |
+
---
|
| 207 |
+
|
| 208 |
+
## 5. Intended use
|
| 209 |
+
|
| 210 |
+
- **Research and demonstration** of a modular, CPU-first remote-sensing question-answering system.
|
| 211 |
+
- **Routing and dispatch** of natural-language queries to the appropriate specialist, using the
|
| 212 |
+
`router` adapter over frozen MiniLM embeddings.
|
| 213 |
+
- **Reproducible evaluation** of each specialist on its own documented split, using the released
|
| 214 |
+
artifacts and the frozen config hash `78f1e3700da15aa1`.
|
| 215 |
+
- **Teaching and ablation**: the six artifacts are small and individually inspectable; the frozen-backbone
|
| 216 |
+
design makes each head a self-contained experiment.
|
| 217 |
+
|
| 218 |
+
The artifacts are intended to be used **with their pinned backbones** (Β§4), which the consumer must
|
| 219 |
+
fetch separately.
|
| 220 |
+
|
| 221 |
+
---
|
| 222 |
+
|
| 223 |
+
## 6. Out-of-scope use
|
| 224 |
|
| 225 |
- **Safety-, legal- or life-critical decisions.** No accuracy, calibration or robustness guarantee is
|
| 226 |
+
offered. Grounding boxes are **image-relative, not geodetic** β no geolocation accuracy
|
| 227 |
+
(`docs/LIMITATIONS.md` Β§7).
|
| 228 |
- **Operational geospatial production** without independent validation.
|
| 229 |
+
- **Any use of the VLM adapter as a production model** β it is `ACCEPTANCE-REJECTED` (Β§10); the deployed
|
| 230 |
+
caption/VQA path uses the **unadapted** model.
|
| 231 |
+
- **Treating per-specialist metrics as system-level accuracy.** No end-to-end benchmark exists (Β§11).
|
| 232 |
+
- **Any claim that these artifacts generalise beyond their training-family test splits** β cross-dataset
|
| 233 |
+
generalisation is `NOT RUN`.
|
| 234 |
+
- **Redistribution of the backbones.** This release contains no backbone weights.
|
| 235 |
+
|
| 236 |
+
---
|
| 237 |
|
| 238 |
+
## 7. Per-artifact reference
|
| 239 |
|
| 240 |
+
Each subsection gives architecture, hyperparameters (from `configs/base.yaml` unless noted), training
|
| 241 |
+
data, evaluation protocol, measured numbers (with source artifact and key path), acceptance status, and
|
| 242 |
+
limitations.
|
| 243 |
+
|
| 244 |
+
### 7.0 Enforced configuration invariants, with arithmetic
|
| 245 |
+
|
| 246 |
+
The frozen registry is `configs/base.yaml`; the project rule is **"no magic numbers anywhere in Python;
|
| 247 |
+
everything tunable lives here"**, and `core/config.py` loads, validates and hashes every value. Several
|
| 248 |
+
values are *enforced* β a mismatch is a load-time error, not a comment:
|
| 249 |
+
|
| 250 |
+
| Invariant | Arithmetic / rule | Why it is enforced |
|
| 251 |
+
|---|---|---|
|
| 252 |
+
| Fusion input width | `3 Γ 768 + 12 + 2 = 2318` | `core/config.py` recomputes it and `fusion_head.py` recomputes it **again**, refusing to build on mismatch β a config edit cannot silently reshape the first `Linear` |
|
| 253 |
+
| Grounding head feature width | `4 Γ 512 = 2048` | `core/config.py` rejects any other value and `specialists/grounding/remoteclip.py` asserts it against the real model β a mismatch is a **silent** shape error otherwise |
|
| 254 |
+
| CROMA resolution | `image_resolution % 8 == 0`; native `120` β `225` patches | required by CROMA (finding C-7) |
|
| 255 |
+
| Router truncation | `max_length β€ 256` | MiniLM tokenizer ceiling; truncating above it is a silent no-op (F4-1) |
|
| 256 |
+
| VLM processor | `processor_longest_edge β€ image.tile_size` | otherwise the processor upscales and splits a tile ~17Γ (F5-2) |
|
| 257 |
+
| Change tile | `256`; `tile_overlap: 0` | STANet-style detector; LEVIR-CD-256 |
|
| 258 |
+
| Frozen config hash | `Config.hash = sha256(base.yaml)[:16] = 78f1e3700da15aa1` | every artifact records it; a config edit detaches the numbers from their configuration |
|
| 259 |
+
|
| 260 |
+
The frozen hash is verified untouched by `test_the_frozen_config_hash_has_not_moved`
|
| 261 |
+
(`docs/OWNER_DECISIONS_2026-09-23.md`, cross-cutting rule 4). New defaults live in **code**, not in the
|
| 262 |
+
registry β which is why the grounding head's default path is `DEFAULT_HEAD_PATH` in code rather than a
|
| 263 |
+
`base.yaml` key (owner decision D-4).
|
| 264 |
+
|
| 265 |
+
---
|
| 266 |
+
|
| 267 |
+
### 7.1 `change` β STANet-style Siamese change detector
|
| 268 |
+
|
| 269 |
+
**Kind:** trained head Β· **File:** `change/head.pt` Β· **Bytes:** 63,231,009 Β· **sha256:**
|
| 270 |
+
`c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa`
|
| 271 |
+
|
| 272 |
+
**Architecture.** A STANet-style Siamese detector: a shared ResNet-18 encoder (`SharedResNetEncoder`), a
|
| 273 |
+
`DifferenceFusion` module per stage, **PAM** spatial self-attention (`SpatialAttention`, `sa_mode: PAM`;
|
| 274 |
+
BAM is the alternative), a three-stage decoder (`dec3 β dec2 β dec1`) with a final upsample and a 1Γ1
|
| 275 |
+
convolution head to one change logit. Source: `specialists/change/stanet.py`. Manifest architecture
|
| 276 |
+
string: `"STANet-style Siamese change detector (ResNet-18 + PAM)"`.
|
| 277 |
+
|
| 278 |
+
**Hyperparameters** (`base.yaml` β `change:`; artifact `checkpoint_embedded_config`):
|
| 279 |
+
|
| 280 |
+
| Parameter | Value | | Parameter | Value |
|
| 281 |
+
|---|---|---|---|---|
|
| 282 |
+
| `tile_size` / `tile_overlap` | 256 / 0 | | `threshold` | 0.50 |
|
| 283 |
+
| `min_component_pixels` | 32 | | `encoder` | `resnet18` |
|
| 284 |
+
| `encoder_channels` | `[64,128,256,512]` | | `width` | 128 |
|
| 285 |
+
| `sa_mode` | `PAM` | | `pretrained` | `true` |
|
| 286 |
+
| `frozen_encoder` | `false` | | `attention_budget_bytes` | 268,435,456 |
|
| 287 |
+
| `learning_rate` / `batch_size` | 0.001 / 8 | | `bce_weight` / `dice_weight` | 0.5 / 0.5 |
|
| 288 |
+
|
| 289 |
+
**Training data.** LEVIR-CD-256, split `train 7,120 / val 1,024 / test 2,048` (`base.yaml` β
|
| 290 |
+
`change.levir_split`), matching the published LEVIR-CD counts exactly
|
| 291 |
+
(`docs/OWNER_DECISIONS_2026-09-23.md` D-11). Trained on GPU (eval artifact: `device: cuda`,
|
| 292 |
+
torch 2.10.0+cu128, python 3.12.13).
|
| 293 |
+
|
| 294 |
+
**Evaluation protocol.** Held-out `test` split, n = 2048, threshold 0.50, tile 256, no overlap. Source:
|
| 295 |
+
`artifacts/change/eval_test/eval_result.json`. The artifact checks the checkpoint's embedded config
|
| 296 |
+
against the frozen hash (`checkpoint_config_hash_checked: true`, `config_drift: false`).
|
| 297 |
+
|
| 298 |
+
**Measured numbers:**
|
| 299 |
+
|
| 300 |
+
| Metric | Value | Key path |
|
| 301 |
+
|---|---|---|
|
| 302 |
+
| **pooled IoU** | **0.8122** | `metrics.pooled.iou` |
|
| 303 |
+
| **macro IoU** | **0.8457** | `metrics.macro.miou` |
|
| 304 |
+
| **pooled F1** | **0.8964** | `metrics.pooled.f1` |
|
| 305 |
+
| pooled `miou` / precision / recall | 0.9007 / 0.9195 / 0.8745 | `metrics.pooled.*` |
|
| 306 |
+
| macro F1 / `iou` / precision / recall | 0.7962 / 0.7180 / 0.8506 / 0.7757 | `metrics.macro.*` |
|
| 307 |
+
| confusion | tp 5,978,997 Β· fp 523,658 Β· fn 858,407 Β· tn 126,856,666 | `metrics.pooled.*` |
|
| 308 |
+
| n pixels | 134,217,728 | `metrics.pooled.n_pixels` |
|
| 309 |
+
| images with change | 935 / 2,048 | `n_images_with_change` |
|
| 310 |
+
| mean change fraction | 0.0509 (p50 0.0, p90 0.197205, max 0.684937) | `metrics.mean_change_fraction`, `change_fraction_quantiles` |
|
| 311 |
+
| wall time | 55.359 s | `metrics.seconds` |
|
| 312 |
+
|
| 313 |
+
**Acceptance status:** **VERIFIED** and accepted (shipped). This is the **only** headline metric in the
|
| 314 |
+
project carrying the `VERIFIED` tag β measured against a single, immutable public test split with a
|
| 315 |
+
frozen threshold (`docs/BENCHMARKS.md` Β§1.2).
|
| 316 |
+
|
| 317 |
+
**Limitations.** Pooled and macro figures diverge (IoU 0.8122 vs 0.8457; F1 0.8964 vs 0.7962), and the
|
| 318 |
+
corpus is heavily zero-change (`p50` change fraction 0.0; only 935 of 2,048 images contain change). No
|
| 319 |
+
cross-dataset evaluation was run.
|
| 320 |
+
|
| 321 |
+
---
|
| 322 |
+
|
| 323 |
+
### 7.2 `change_vqa` β change question answering head
|
| 324 |
+
|
| 325 |
+
**Kind:** trained head Β· **File:** `change_vqa/head.pt` Β· **Bytes:** 5,822,809 Β· **sha256:**
|
| 326 |
+
`cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` Β· **Parameters:** 1,453,912
|
| 327 |
+
|
| 328 |
+
**Architecture.** `change_vqa_head_v1` β a **two-stage** reasoning head, not a generative decoder
|
| 329 |
+
(`training/change_vqa/model.py`). Stage 1 maps the change representation to a **class-wise change
|
| 330 |
+
estimate** β 6 magnitudes + 6 signed deltas + 1 global fraction = **13 outputs**
|
| 331 |
+
(`N_ESTIMATOR_OUTPUTS = 2 Γ N_CHANGE_CLASSES + 1`), supervised by `label1`/`label2`. Stage 2
|
| 332 |
+
concatenates that estimate with a question encoding and predicts one of **19 answers** (`N_ANSWERS`).
|
| 333 |
+
The estimator's outputs are *also* emitted as evidence, so an answer arrives with its own audit trail.
|
| 334 |
+
Modules: an `estimator` MLP (`change_feature_dim β 256 β 13`), a `change_trunk`, a `question_trunk`
|
| 335 |
+
(text feature + question-type embedding + temporal embedding) and an `answer_head`
|
| 336 |
+
(`fused β 512 β 256 β 19`).
|
| 337 |
+
|
| 338 |
+
**Hyperparameters** (module constants; `base.yaml` β `training:` where applicable):
|
| 339 |
+
|
| 340 |
+
| Parameter | Value | | Parameter | Value |
|
| 341 |
+
|---|---|---|---|---|
|
| 342 |
+
| `ARCHITECTURE_VERSION` | `change_vqa_head_v1` | | `trunk_dim` / `text_dim` | 512 / 256 |
|
| 343 |
+
| `dropout` | 0.10 | | `qtype_embed_dim` / `temporal_embed_dim` | 32 / 8 |
|
| 344 |
+
| estimator outputs | 13 (2 Γ 6 + 1) | | answer space | 19 |
|
| 345 |
+
| `seed` | 42 | | `epoch_selected` | 8 (on Val answer accuracy) |
|
| 346 |
+
| `stop_reason` | `early_stopping` | | | |
|
| 347 |
+
|
| 348 |
+
**Training data.** CDVQA (`dataset_id: cdvqa`), feature specs `change_feat_v1`,
|
| 349 |
+
`change_cache_spec: c801326f85a185f8`, `text_cache_spec: d2801ea1a314354a`,
|
| 350 |
+
`preprocessing_version: change_vqa_preproc_v1`. The head's change features are backed by the **frozen**
|
| 351 |
+
STANet change detector β `frozen_dependency.path: artifacts/change/levir_change_v001/head.pt`, sha256
|
| 352 |
+
`c5ef31277b67aa01β¦`, `verified_byte_exact_vs_local: true` (the *same* artifact as Β§7.1). Trained on an
|
| 353 |
+
external GPU (Kaggle) β see Β§13.
|
| 354 |
+
|
| 355 |
+
**Evaluation protocol.** Two held-out test sets, `test` (n = 39,686) and `test2` (n = 31,036).
|
| 356 |
+
`epoch_selected` was chosen on **Val answer accuracy = 0.700018**; `PROMOTION.json` records 93 checks
|
| 357 |
+
passed, 0 failed, 0 unverified. Source: `artifacts/change_vqa/run/PROMOTION.json` β `verification`.
|
| 358 |
+
|
| 359 |
+
**Measured numbers:**
|
| 360 |
+
|
| 361 |
+
| Metric | `test` | `test2` |
|
| 362 |
+
|---|---|---|
|
| 363 |
+
| **accuracy** | **0.697626367** | **0.651469262** |
|
| 364 |
+
| **macro F1** | **0.378373275** | **0.372308516** |
|
| 365 |
+
| global-majority baseline | 0.311546 | 0.178728 |
|
| 366 |
+
| n scored | 39,686 | 31,036 |
|
| 367 |
+
|
| 368 |
+
`mask_gain: 0.0`. `metric_ruling: "OPEN β the plan leaves the accuracy/macro-F1 interpretation
|
| 369 |
+
owner-gated. No official aggregate metric is asserted here."`
|
| 370 |
+
|
| 371 |
+
**Acceptance status:** MEASURED on two test sets; **ruling OPEN**. `PROMOTION.json` is explicit that
|
| 372 |
+
promotion "records provenance and wires the serving path. It does not itself confer VERIFIED status."
|
| 373 |
+
|
| 374 |
+
**Limitations.** The wide accuracyβmacro-F1 gap (0.697626 vs 0.378373) is the signature of class
|
| 375 |
+
imbalance: accuracy is dominated by frequent answers while macro-F1 exposes weak rare-class performance
|
| 376 |
+
(`docs/LIMITATIONS.md` Β§1.4). Confidence at this head is **raw, not calibrated** (`method` reads
|
| 377 |
+
`"uncalibrated"`). The two test sets disagree (0.697626 vs 0.651469), so quoting one alone is selective.
|
| 378 |
+
|
| 379 |
+
---
|
| 380 |
+
|
| 381 |
+
### 7.3 `optical_sar` β CROMA-base fusion head
|
| 382 |
+
|
| 383 |
+
**Kind:** trained head (production) Β· **File:** `optical_sar/head.pt` Β· **Bytes:** 14,427,457 Β·
|
| 384 |
+
**sha256:** `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab`
|
| 385 |
+
|
| 386 |
+
**Architecture.** A fusion head over **frozen CROMA-base** features. CROMA emits three 768-d GAP vectors
|
| 387 |
+
per sample (`optical_GAP`, `SAR_GAP`, `joint_GAP`); the head concatenates them with the availability
|
| 388 |
+
masks β `optical_mask (B,12)` and `sar_mask (B,2)` β into a **(B, 2318)** tensor (`3Γ768 + 12 + 2`), then
|
| 389 |
+
`LayerNorm β Linear(2318 β 512) β GELU β Dropout(0.2) β Linear(512 β 19)`. Source:
|
| 390 |
+
`specialists/optical_sar/fusion_head.py`. The **availability mask is consumed by the head, not by
|
| 391 |
+
CROMA** (finding C-1): handing CROMA the mask would invite it to reconstruct missing channels β the
|
| 392 |
+
fabrication the sensor adapter exists to prevent.
|
| 393 |
+
|
| 394 |
+
**Hyperparameters** (`base.yaml` β `croma:` and `fusion:`):
|
| 395 |
+
|
| 396 |
+
| Parameter | Value | | Parameter | Value |
|
| 397 |
+
|---|---|---|---|---|
|
| 398 |
+
| `croma.checkpoint_file` | `CROMA_base.pt` (rev `0dd28e3d633b`) | | `croma.image_resolution` | 120 (`% 8 == 0`) |
|
| 399 |
+
| `croma.encoder_dim` | 768 | | `croma.optical_channels` / `sar_channels` | 12 / 2 |
|
| 400 |
+
| `croma.modalities_used` | `[optical, sar, joint]` | | `fusion.input_dim` | 2318 |
|
| 401 |
+
| `fusion.hidden_dim` | 512 | | `fusion.dropout` | 0.2 |
|
| 402 |
+
| `fusion.num_classes` | 19 (BigEarthNet CLC) | | channel/band dropout | **mandatory** (freeze Β§2.5) |
|
| 403 |
+
|
| 404 |
+
**Training data.** reBEN / BigEarthNet-S1 (`data/bigearthnet_v2/`, 480,038 rows in `metadata.parquet`;
|
| 405 |
+
`docs/OWNER_DECISIONS_2026-09-23.md` D-11). The extraction used the **`require_single_label` policy**
|
| 406 |
+
(`n_skipped_by_policy: 0`), which preserves the frozen single-label 19-class softmax but **changes the
|
| 407 |
+
evaluation population** (see limitations). The A/B arm decision was made separately on
|
| 408 |
+
`best_val_accuracy` β **A 0.837100 vs B 0.839100**, floor 0.0285 β **Arm A retained** (owner ruling
|
| 409 |
+
R-14; `docs/PHASE12_115_METRIC_COMPUTED.md` Β§5).
|
| 410 |
+
|
| 411 |
+
**Evaluation protocol.** The pre-registered 11.5 metric: fusion-head accuracy and macro-F1 over the
|
| 412 |
+
19-class label space on the held-out `test` split, n = 4,000, cache arm A
|
| 413 |
+
(`docs/PHASE14_CROMA_NORMALISATION_CHANGE.md` Β§4). Computed by a **separate, later, read-only** tool
|
| 414 |
+
(`scripts/eval_fusion_115.py`); the trainer deliberately never opens the test split
|
| 415 |
+
(`pre_registered_metric_computed = false` in every run record). Source:
|
| 416 |
+
`artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`.
|
| 417 |
+
|
| 418 |
+
**Measured numbers:**
|
| 419 |
+
|
| 420 |
+
| Metric | Value | Key path |
|
| 421 |
+
|---|---|---|
|
| 422 |
+
| **accuracy** | **0.931** | `accuracy` |
|
| 423 |
+
| **macro F1** | **0.434161** | `macro_f1` |
|
| 424 |
+
| loss | 0.254592 | `loss` |
|
| 425 |
+
| n scored / classes | 4,000 / 19 | `n_scored` / `num_classes` |
|
| 426 |
+
| classes present | `[0,2,3,4,5,6,7,8,9,10,12,13,17,18]` | `classes_present` |
|
| 427 |
+
| classes absent | `[1,11,14,15,16]` | `classes_absent` |
|
| 428 |
+
| macro-F1 denominator | all 19 slots (absent classes contribute 0.0) | `macro_f1_denominator` |
|
| 429 |
+
| present-only macro-F1 (diagnostic) | 0.589218 | `docs/PHASE12_115_METRIC_COMPUTED.md` Β§3.5 |
|
| 430 |
+
|
| 431 |
+
The majority class holds 2,264 / 4,000 = **0.566**, so 0.931 is not a constant predictor. Per-class F1
|
| 432 |
+
(`_per_class_f1`) shows a wide spread: one class is perfect (1.000), while classes **5** and **6** are
|
| 433 |
+
**present but score 0.000** β genuine per-class failures, not absent-class artifacts. The median of the
|
| 434 |
+
14 present classes is **0.6857** against an accuracy of 0.931 β the signature of prediction dominated by
|
| 435 |
+
frequent classes (`docs/PHASE12_115_METRIC_COMPUTED.md` Β§3.4βΒ§3.5).
|
| 436 |
+
|
| 437 |
+
**Acceptance status:** MEASURED; **ruling OPEN**. Whether 0.931/0.434 constitutes a Phase 12 *pass* is
|
| 438 |
+
the **metric-of-record ruling**, which "has not been made, and it is not engineering's to make." The
|
| 439 |
+
artifact's own `is_deciding_statistic: false` and advisory text state that it "selects no head, ranks
|
| 440 |
+
nothing and compares no arms."
|
| 441 |
+
|
| 442 |
+
**Limitations.** (1) **Never quote accuracy alone** β 0.931 travels with macro-F1 0.434161. (2) The
|
| 443 |
+
metric describes a **single-label subset**, not multi-label reBEN: single-label patches are 17.57 % of
|
| 444 |
+
the corpus (96,537 / 549,488), and under this policy the rarest class survives as **1 patch** (a
|
| 445 |
+
59,204 : 1 imbalance). It **may not** be presented as a multi-label BigEarthNet/reBEN result, nor as
|
| 446 |
+
comparable to published BigEarthNet numbers, nor as a statement about all 19 classes β 5 have no test
|
| 447 |
+
samples here. (3) The live service returns a **bare class index** (`class_18`), not a CLC label
|
| 448 |
+
(`docs/LIMITATIONS.md` Β§1.6).
|
| 449 |
|
| 450 |
+
---
|
| 451 |
+
|
| 452 |
+
### 7.4 `grounding` β RemoteCLIP grounding head
|
| 453 |
+
|
| 454 |
+
**Kind:** trained head Β· **File:** `grounding/head.pt` Β· **Bytes:** 12,639,041 Β· **sha256:**
|
| 455 |
+
`93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb` Β· **Parameters:** 1,052,677
|
| 456 |
+
(`docs/OWNER_DECISIONS_2026-09-23.md` D-4, measured against the real checkpoint).
|
| 457 |
+
|
| 458 |
+
**Architecture.** A text-conditioned per-cell box regressor over **frozen RemoteCLIP ViT-B/32** tokens.
|
| 459 |
+
At 224 px the patch grid is **7Γ7 = 49** tokens of projected dim **512**; the text embedding (512) is
|
| 460 |
+
broadcast to every cell, and each cell's feature is `concat([patch, text, patch*text, global_pool]) =
|
| 461 |
+
4 Γ 512 = 2048`. The head is `Linear(2048 β 512) β LayerNorm β Dropout(0.10) β Linear(512 β 5)`,
|
| 462 |
+
emitting `[tx, ty, tw, th, obj]` per cell. Boxes are **cell-relative** (YOLO-style), and exactly **one
|
| 463 |
+
cell per target** is positive β the one containing the ground-truth box centre. Source:
|
| 464 |
+
`specialists/grounding/head.py`.
|
| 465 |
+
|
| 466 |
+
**Hyperparameters** (`base.yaml` β `grounding:`, `grounding_head:`, `grounding_training:`):
|
| 467 |
+
|
| 468 |
+
| Parameter | Value | | Parameter | Value |
|
| 469 |
+
|---|---|---|---|---|
|
| 470 |
+
| `image_size` | 224 (`resolution_frozen: true`) | | `model_name` | `ViT-B-32` |
|
| 471 |
+
| `encoder_projected_dim` | 512 (width 768 β projected 512, P7-1) | | `nms_iou` | 0.50 |
|
| 472 |
+
| `max_candidates` | 20 | | `confidence_threshold` | 0.40 |
|
| 473 |
+
| `benchmark_box_scale` | 100.0 (VRSBench 0β100 β stored 0β1) | | `head.feature_dim` | 2048 |
|
| 474 |
+
| `head.hidden_dim` | 512 | | `head.dropout` | 0.10 |
|
| 475 |
+
| `head.positive_confidence_weight` | 20.0 (1 positive in 49) | | `head.decode` | `cell_relative` |
|
| 476 |
+
| training lr / batch / epochs | 1e-4 / 16 / 20 | | training wd / warmup / grad_clip | 1e-4 / 0.05 / 1.0 |
|
| 477 |
+
| training `val_fraction` | 0.10 | | loss weights box/giou/conf | 0.5 / 0.3 / 0.2 |
|
| 478 |
+
|
| 479 |
+
**Training data.** VRSBench (`training/data/vrsbench/`), 16,159 eval records, all images present
|
| 480 |
+
(`docs/OWNER_DECISIONS_2026-09-23.md` D-11). Resolution frozen at **224** by a pre-registered decision
|
| 481 |
+
(see protocol).
|
| 482 |
+
|
| 483 |
+
**Evaluation protocol.** Full VRSBench eval split, **16,159 / 16,159 records**, resolution 224, CPU
|
| 484 |
+
(canonical and matched6 artifacts record `device: cpu`, torch 2.14.0+cpu). Grounding is reported under
|
| 485 |
+
**two protocols** β *canonical* (config default `top_k = 20`) and *matched6* (`top_k = 6`, matching the
|
| 486 |
+
zero-shot baseline's mean 5.99 candidates) β and **two decode variants** β `head_threshold` (score
|
| 487 |
+
threshold 0.40) and `head_argmax`. `head_decode`: `nms_iou 0.5`, `score_threshold 0.4`.
|
| 488 |
+
|
| 489 |
+
**Measured numbers** (canonical: `β¦/eval_result_canonical.json`; matched6: `β¦/eval_result_matched6.json`):
|
| 490 |
+
|
| 491 |
+
| Protocol / decode | mean best IoU | recall@0.10 | recall@0.25 | recall@0.50 |
|
| 492 |
+
|---|---|---|---|---|
|
| 493 |
+
| **canonical** `head_threshold` | **0.2838** | 0.6882 | 0.5047 | **0.2198** |
|
| 494 |
+
| **canonical** `head_argmax` | **0.1215** | 0.3183 | 0.2088 | 0.0795 |
|
| 495 |
+
| **canonical** `zero_shot_matched` | **0.0972** | 0.3298 | 0.1188 | 0.0234 |
|
| 496 |
+
| **matched6** `head_threshold` | **0.2566** | 0.6315 | 0.4545 | **0.1938** |
|
| 497 |
+
| **matched6** `head_argmax` | **0.1215** | 0.3183 | 0.2088 | 0.0795 |
|
| 498 |
+
| **matched6** `zero_shot_matched` | **0.0972** | 0.3298 | 0.1188 | 0.0234 |
|
| 499 |
+
|
| 500 |
+
Latency: `head_threshold` 2.205 ms/image (canonical) / 2.158 (matched6); `head_argmax` 0.655 / 0.652;
|
| 501 |
+
`zero_shot_matched` 17.9 s / 15.5 s total. The zero-shot decode is
|
| 502 |
+
`threshold_box_plus_local_maxima`, `delta 0.02`, `top_k 5`, mean 5.99 candidates/image. The artifact's
|
| 503 |
+
`phase7_reference` records the zero-shot floor `mean_best_iou 0.0972`, `recall_at_0.50 0.0234`
|
| 504 |
+
(`docs/PHASE7_RESOLUTION_DECISION.md`).
|
| 505 |
+
|
| 506 |
+
**The 224-vs-448 decision (pre-registered, then confirmed).** The rule was fixed *before* the result was
|
| 507 |
+
seen: *448 wins if Recall@0.5 improves by β₯ 0.05 absolute OR mean best IoU improves by β₯ 0.05 absolute;
|
| 508 |
+
224 wins otherwise.* Result: **224 WINS** β mean best IoU gain β0.0147, recall@0.5 gain β0.0022, at
|
| 509 |
+
1.59Γ the latency. Paired over the identical 16,159 samples: mean paired diff β0.0147, 95 % CI
|
| 510 |
+
[β0.0160, β0.0134], **t = β22.63**; 448 better on 8.5 %, worse on **20.9 %**. The artifact records
|
| 511 |
+
`rule_changed_since_preregistration: false` (`docs/PHASE7_RESOLUTION_DECISION.md`).
|
| 512 |
+
|
| 513 |
+
**Acceptance status:** MEASURED under two protocols; shipped. The trained head is the **production
|
| 514 |
+
default** (owner decision D-4); zero-shot is an **explicit, labelled fallback**, and the system must
|
| 515 |
+
never silently claim `trained` while running zero-shot.
|
| 516 |
+
|
| 517 |
+
**Limitations.** Absolute IoU is low (0.2838 canonical / 0.2566 matched6) β the head clearly beats the
|
| 518 |
+
zero-shot baseline (0.0972) but 0.28 is not "solved". The number is **protocol-sensitive**: an absolute
|
| 519 |
+
value is meaningless without its protocol and decode variant. `head_argmax` (0.1215) is **not**
|
| 520 |
+
apples-to-apples with the multi-box baseline (mean best IoU is a max over predictions, so 1 box vs ~6
|
| 521 |
+
flatters the head). Boxes are image-relative, not geodetic. 448 was rejected at the *zero-shot* level;
|
| 522 |
+
whether a *learned* head has the same resolution sensitivity is `UNKNOWN β not established from the
|
| 523 |
+
available evidence` (`docs/PHASE7_RESOLUTION_DECISION.md`).
|
| 524 |
+
|
| 525 |
+
---
|
| 526 |
+
|
| 527 |
+
### 7.5 `router` β intent adapter over frozen MiniLM
|
| 528 |
+
|
| 529 |
+
**Kind:** trained adapter Β· **File:** `router/adapter.pt` Β· **Bytes:** 211,961 Β· **sha256:**
|
| 530 |
+
`8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` Β· **Parameters:** ~50,822
|
| 531 |
+
|
| 532 |
+
**Architecture.** The only trainable part of the router (`router/adapter.py`, `IntentAdapter`):
|
| 533 |
+
|
| 534 |
+
```
|
| 535 |
+
embedding (384) β LayerNorm β Linear(384 β 128) β GELU β Dropout(0.10)
|
| 536 |
+
βββ task_head Linear(128 β 6) # vqa/caption/grounding/change/optical_sar/unsupported
|
| 537 |
+
βββ modality_head Linear(128 β 4)
|
| 538 |
+
βββ temporal_head Linear(128 β 1) # logit; P(yes) = sigmoid(logit)
|
| 539 |
+
βββ spatial_head Linear(128 β 1)
|
| 540 |
+
βββ language_head Linear(128 β 1)
|
| 541 |
+
```
|
| 542 |
+
|
| 543 |
+
Heads are initialised with small-std weights (`std 0.02`, zero bias) so the initial sigmoid sits near
|
| 544 |
+
0.5 and the binary heads do not start saturated. The adapter does **not** back-propagate into MiniLM.
|
| 545 |
+
|
| 546 |
+
**Hyperparameters** (`base.yaml` β `router:` and `router.training:`):
|
| 547 |
+
|
| 548 |
+
| Parameter | Value | | Parameter | Value |
|
| 549 |
+
|---|---|---|---|---|
|
| 550 |
+
| `model` | `all-MiniLM-L6-v2` (rev `1110a243fdf4`) | | `max_length` | 128 (ceiling is 256) |
|
| 551 |
+
| `embedding_dim` | 384 | | `hidden_dim` | 128 |
|
| 552 |
+
| `dropout` | 0.10 | | `num_tasks` | 6 |
|
| 553 |
+
| `confidence_threshold` | 0.70 | | `epochs` / `batch_size` | 60 / 64 |
|
| 554 |
+
| `learning_rate` / `weight_decay` | 0.001 / 0.01 | | loss weights task/modality/binary | 1.0 / 0.3 / 0.5 |
|
| 555 |
+
| `val_ratio` | 0.15 | | `hard_negatives_to_test` | `true` |
|
| 556 |
+
|
| 557 |
+
**Training data.** A synthetic query corpus: **576** queries in **54** groups
|
| 558 |
+
(`artifacts/router/threshold_sweep_val.json` β `corpus_total`, `corpus_groups`), split
|
| 559 |
+
`train 410 / val 86 / test 80`. Splits are **by group** (template / hard-negative family), never by
|
| 560 |
+
example, so template variants cannot leak across the boundary (F4-3). Hard-negative families are placed
|
| 561 |
+
in the **test** split so their accuracy measures generalisation, not memorisation. The encoder is frozen,
|
| 562 |
+
so embeddings are cached and the adapter trains on cached vectors β measured on CPU: **20 epochs over
|
| 563 |
+
4,096 Γ 384 in 0.28 s** (F4-2).
|
| 564 |
+
|
| 565 |
+
**Evaluation protocol.** A **validation-only threshold sweep** over 50 thresholds `0.50 β¦ 0.99`, on val
|
| 566 |
+
**n = 86**, `select_by: covered_accuracy`. Source: `artifacts/router/threshold_sweep_val.json`. The test
|
| 567 |
+
split was **NOT touched** (`test_split_touched: false`, `n_test_examples_scored: 0`).
|
| 568 |
+
|
| 569 |
+
**Measured numbers:**
|
| 570 |
|
| 571 |
+
| Metric | Value | Key path |
|
| 572 |
+
|---|---|---|
|
| 573 |
+
| **overall ungated accuracy** | **0.965116** | `overall_ungated_accuracy` |
|
| 574 |
+
| n val | 86 | `n_val` |
|
| 575 |
+
| corpus total / groups | 576 / 54 | `corpus_total` / `corpus_groups` |
|
| 576 |
+
| split sizes | train 410 / val 86 / test 80 | `split_sizes` |
|
| 577 |
+
| shipped threshold | 0.70 | `shipped_threshold` |
|
| 578 |
+
| shipped row (thr 0.70) | coverage 0.848837 Β· covered acc 0.972603 Β· fallback 0.151163 Β· n_covered 73 | `shipped_row` |
|
| 579 |
+
| selected row (thr 0.76) | coverage 0.790698 Β· covered acc 1.0 Β· fallback 0.209302 Β· n_covered 68 | `selected` |
|
| 580 |
+
| val per-task support | caption 8 Β· change 20 Β· grounding 14 Β· optical_sar 10 Β· unsupported 19 Β· vqa 15 | `val_task_counts` |
|
| 581 |
+
| hard negatives in val | 0 | `hard_negatives_in_val` |
|
| 582 |
+
| adapter config hash | `615478910dc266bf` | `adapter_config_hash` |
|
| 583 |
+
| encoder | 22,713,216 params, `max_length` 128, rev `1110a243fdf4` | `adapter_encoder` |
|
| 584 |
+
|
| 585 |
+
**Acceptance status:** MEASURED (val only); shipped; **test split NOT RUN**.
|
| 586 |
+
|
| 587 |
+
**Limitations.** The artifact is explicit that this is **not** a calibration and **not** a test result:
|
| 588 |
+
*"corpus-limited: val n=86 vs plan >=500. This is NOT a calibration β the corpus is synthetic and too
|
| 589 |
+
small (min per-class support 8, caption) and val carries 0 hard negatives (hn_* families are held out to
|
| 590 |
+
TEST by design). Selecting a threshold here yields a justified default, not a calibrated value."*
|
| 591 |
+
`plan_min_val_queries: 500` and `plan_min_hard_negatives: 100` are both unmet. The number is
|
| 592 |
+
**ungated** accuracy, and the router has known residuals β e.g. *"What is the new runway?"* reads
|
| 593 |
+
`change`, not `vqa` (`docs/LIMITATIONS.md` Β§2).
|
| 594 |
+
|
| 595 |
+
---
|
| 596 |
+
|
| 597 |
+
### 7.6 `vlm` β SmolVLM LoRA adapter (USABLE_VERIFIED, ACCEPTANCE-REJECTED)
|
| 598 |
+
|
| 599 |
+
**Kind:** LoRA adapter (PEFT) Β· **File:** `vlm/adapter_model.safetensors` Β· **Bytes:** 34,798,048 Β·
|
| 600 |
+
**sha256:** `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` Β· **Trainable params:**
|
| 601 |
+
8,683,520 (1.6823 % of the 516,165,824-param base)
|
| 602 |
+
|
| 603 |
+
**Architecture.** A **PEFT LoRA** adapter (`r = 16`, `alpha = 32`, `dropout = 0.05`) on the
|
| 604 |
+
**text-model projections** of frozen `HuggingFaceTB/SmolVLM-500M-Instruct` (rev `a7da5b986cb5`). Target
|
| 605 |
+
modules: `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` across **224** modules.
|
| 606 |
+
`trainable_subtrees` is exactly `{"model.text_model": 8683520}` β the **vision tower was untouched**
|
| 607 |
+
(86,433,024 frozen) and the connector (11,796,480) is frozen too. Precision: **fp16**.
|
| 608 |
+
|
| 609 |
+
**Hyperparameters** (`base.yaml` β `training:`, `vlm:`; `phase6_closure.json`):
|
| 610 |
+
|
| 611 |
+
| Parameter | Value | | Parameter | Value |
|
| 612 |
+
|---|---|---|---|---|
|
| 613 |
+
| `checkpoint` | `SmolVLM-500M-Instruct` (rev `a7da5b986cb5`) | | `lora_rank`/`alpha`/`dropout` | 16 / 32 / 0.05 |
|
| 614 |
+
| LoRA target modules | 224 | | `precision` | fp16 (T4 is SM 7.5 β not bf16; C-6) |
|
| 615 |
+
| `vlm_batch_size` / `grad_accum` | 2 / 8 | | `vlm_learning_rate` / `vlm_epochs` | 2e-4 / 1 |
|
| 616 |
+
| `weight_decay` / `warmup_ratio` | 0.01 / 0.05 | | `gradient_checkpointing` | `true` |
|
| 617 |
+
| `processor_longest_edge` | 512 (default 2048 splits a tile into 17 sub-images β F5-2) | | `do_sample` / `temperature` | `false` / 0.0 |
|
| 618 |
+
| `max_new_tokens` | 128 | | `max_images_per_call` / `seed` | 1 / 42 |
|
| 619 |
|
| 620 |
+
**Training data.** BigEarthNet-derived presence questions (`kind: bigearthnet_smolvlm_lora`). Trained on
|
| 621 |
+
an external GPU (Kaggle, T4) β see Β§13. The adapter directory holds **14 files** verified against its
|
| 622 |
+
own `ARTIFACT_SHA256SUMS.json`.
|
| 623 |
|
| 624 |
+
**Evaluation protocol.** A frozen **1,000-question test subset** (`available_per_split {val: 6750,
|
| 625 |
+
test: 7772}`, subset n = 1000, 19 classes summing to 1000). The pre-registered acceptance rule is
|
| 626 |
+
**v002**: V1 requires aggregate test delta β₯ +5.00 pp; V2 (a per-class guardrail) fails a class with
|
| 627 |
+
`n β₯ 20` questions iff it **both** lost β₯ 4 questions **and** has `z β₯ 1.96`. Source:
|
| 628 |
+
`artifacts/vlm/phase6_closure.json` β `why_acceptance_rejected`, `why_usable_verified`.
|
| 629 |
|
| 630 |
+
**Measured numbers:**
|
| 631 |
+
|
| 632 |
+
| Metric | Value | Key path |
|
| 633 |
|---|---|---|
|
| 634 |
+
| **exact_match** | **0.963** | `why_usable_verified.adapted_test.exact_match` |
|
| 635 |
+
| **F1** | **0.96432** | `why_usable_verified.adapted_test.f1` |
|
| 636 |
+
| precision / recall | 0.963391 / 0.965251 | `β¦adapted_test.precision` / `.recall` |
|
| 637 |
+
| confusion | tp 500 Β· fp 19 Β· tn 463 Β· fn 18 | `β¦adapted_test.confusion` |
|
| 638 |
+
| n | 1,000 | `β¦adapted_test.n` |
|
| 639 |
+
| aggregate test delta | **+49.50 pp** (46.80 β 96.30) | `why_usable_verified.aggregate_test_delta_pp` |
|
| 640 |
+
|
| 641 |
+
**Why it is *usable and verified*.** Gate D reproduced Run 1's adapted-test control **exactly**
|
| 642 |
+
(`exact_match 0.963`, `f1 0.9643201542912246`, identical confusion), proving the local artifact *is*
|
| 643 |
+
Run 1's adapter and that CPU/fp32 reproduces the Kaggle T4 endpoint. Gate Aβ³ proved subset identity
|
| 644 |
+
without a model. The 14-file manifest check is **clean** (0 missing, 0 mismatched, 0 extra) and the
|
| 645 |
+
adapter **loads through the production path** (`PeftModel.from_pretrained`).
|
| 646 |
+
|
| 647 |
+
**Why it is *acceptance-rejected*.** V1 **passes** (+49.50 pp β₯ +5.00), but V2 **fails**: class
|
| 648 |
+
**Mixed forest** (`n = 33`) goes `100.00 β 87.8788 pp`, a drop of **12.1212 pp**, `lost_questions 4`,
|
| 649 |
+
`z 2.1335` β failing **both** halves of v002. Per item V, a complete run that fails V2 is `REJECTED`.
|
| 650 |
+
The rejection is narrow (1 of 19 classes fails; 11 improved, 5 held) and is **not a split artefact** β
|
| 651 |
+
the same class also degraded on val in Run 1 (drop 6.4516 pp, n = 31). Residual risk, reported not
|
| 652 |
+
resolved: the verdict rests on 4 questions in one class of 33, the unfloored minimum-size exposure
|
| 653 |
+
recorded at `PHASE6_AUDIT_AND_CONTRACT.md` Β§8.6.
|
| 654 |
+
|
| 655 |
+
**Acceptance status:** **USABLE_VERIFIED** and **ACCEPTANCE-REJECTED** β both true, answering different
|
| 656 |
+
questions. **`USABLE_VERIFIED` β `ACCEPTANCE-ACCEPTED`.** The deployed caption/VQA path uses the
|
| 657 |
+
**unadapted** model; the adapter is enabled only via the `SATQUERY_VLM_ADAPTER` environment variable
|
| 658 |
+
(`specialists/vqa/model.py` β `ADAPTER_ENV_VAR`).
|
| 659 |
+
|
| 660 |
+
**Limitations and traps.** (1) The adapter is **not** accepted for production use. (2) The
|
| 661 |
+
`Mixed forest` regression is **not resolved**. (3) `adapter_sha256` names **two different values** and
|
| 662 |
+
they are not interchangeable β a **tree hash** over the weight map (`5c6b8631β¦`, from
|
| 663 |
+
`training/vlm/artifact.py`) versus the **file** sha256 of `adapter_model.safetensors` (`07c76a75β¦`, from
|
| 664 |
+
`specialists/vqa/model.py::_adapter_sha256`); comparing one against the other produces a false
|
| 665 |
+
"artifact was altered" conclusion. (4) The promoted adapter is **not** `checkpoint-2000` β the three
|
| 666 |
+
weight files have three distinct digests (top-level `07c76a75β¦`, `checkpoint-1500` `7273588eβ¦`,
|
| 667 |
+
`checkpoint-2000` `bf249943β¦`). (5) The adapter's canonical path is under `.scratch/`; it is
|
| 668 |
+
**reconstructible** from `phase6_realbundle.zip` and verified against the two digests above.
|
| 669 |
+
|
| 670 |
+
---
|
| 671 |
+
|
| 672 |
+
## 8. Full measured-performance table
|
| 673 |
+
|
| 674 |
+
Every row names its source artifact and the exact key path. The `n` and `split` columns are part of the
|
| 675 |
+
claim, not decoration: a metric without its population is not a result. All 20 numeric claims are
|
| 676 |
+
checked against these files by [`tools/verify_readme_metrics.py`](tools/verify_readme_metrics.py); its
|
| 677 |
+
output (`ALL CLAIMS VERIFIED`) is committed as
|
| 678 |
+
[`tools/readme_metrics_report.txt`](tools/readme_metrics_report.txt).
|
| 679 |
+
|
| 680 |
+
| Capability | Metric | Value | Split / protocol | n | Source β key path | Status |
|
| 681 |
+
|---|---|---|---|---|---|---|
|
| 682 |
+
| Change | pooled IoU | **0.8122** | LEVIR-CD-256 test, thr 0.50 | 2,048 | `change/eval_test/eval_result.json` β `metrics.pooled.iou` | **VERIFIED** |
|
| 683 |
+
| Change | macro IoU | **0.8457** | same | 2,048 | `β¦` β `metrics.macro.miou` | **VERIFIED** |
|
| 684 |
+
| Change | pooled F1 | **0.8964** | same | 2,048 | `β¦` β `metrics.pooled.f1` | **VERIFIED** |
|
| 685 |
+
| Grounding | mean best IoU | **0.2838** | VRSBench canonical (thr, top_k 20) | 16,159 | `grounding/β¦/eval_result_canonical.json` β `results.head_threshold.mean_best_iou` | MEASURED (2 protocols) |
|
| 686 |
+
| Grounding | recall@0.5 | **0.2198** | canonical | 16,159 | `β¦canonical.json` β `results.head_threshold.recall.0.50` | MEASURED (2 protocols) |
|
| 687 |
+
| Grounding | mean best IoU | **0.2566** | VRSBench matched6 (thr, top_k 6) | 16,159 | `β¦matched6.json` β `results.head_threshold.mean_best_iou` | MEASURED (2 protocols) |
|
| 688 |
+
| Grounding | recall@0.5 | **0.1938** | matched6 | 16,159 | `β¦matched6.json` β `results.head_threshold.recall.0.50` | MEASURED (2 protocols) |
|
| 689 |
+
| Grounding | head-argmax IoU | **0.1215** | canonical (argmax) | 16,159 | `β¦canonical.json` β `results.head_argmax.mean_best_iou` | MEASURED |
|
| 690 |
+
| Grounding | zero-shot baseline IoU | **0.0972** | canonical (no head) | 16,159 | `β¦canonical.json` β `results.zero_shot_matched.mean_best_iou` | MEASURED (baseline) |
|
| 691 |
+
| Optical-SAR | accuracy | **0.931** | held-out test, 19 classes | 4,000 | `optical_sar/β¦/pre_registered_115_metric.json` β `accuracy` | MEASURED, ruling **OPEN** |
|
| 692 |
+
| Optical-SAR | macro F1 | **0.434161** | same | 4,000 | `β¦` β `macro_f1` | MEASURED, ruling **OPEN** |
|
| 693 |
+
| Change-VQA | accuracy | **0.697626** | `test` | 39,686 | `change_vqa/run/PROMOTION.json` β `verification.test_accuracy` | MEASURED, ruling **OPEN** |
|
| 694 |
+
| Change-VQA | macro F1 | **0.378373** | `test` | 39,686 | `β¦PROMOTION.json` β `verification.test_macro_f1` | MEASURED, ruling **OPEN** |
|
| 695 |
+
| Change-VQA | accuracy (2nd set) | **0.651469** | `test2` | 31,036 | `β¦PROMOTION.json` β `verification.test2_accuracy` | MEASURED, ruling **OPEN** |
|
| 696 |
+
| Change-VQA | macro F1 (2nd set) | **0.372309** | `test2` | 31,036 | `β¦PROMOTION.json` β `verification.test2_macro_f1` | MEASURED, ruling **OPEN** |
|
| 697 |
+
| VLM (adapted) | exact_match | **0.963** | frozen 1,000-question subset | 1,000 | `vlm/phase6_closure.json` β `why_usable_verified.adapted_test.exact_match` | MEASURED, **ACCEPTANCE-REJECTED** |
|
| 698 |
+
| VLM (adapted) | F1 | **0.96432** | same | 1,000 | `β¦phase6_closure.json` β `β¦adapted_test.f1` | MEASURED, **ACCEPTANCE-REJECTED** |
|
| 699 |
+
| Router | overall **ungated** accuracy | **0.965116** | val, corpus-limited | 86 | `router/threshold_sweep_val.json` β `overall_ungated_accuracy` | MEASURED β **TEST NOT RUN** |
|
| 700 |
+
| Calibration | ECE before / after | **0.013755 β 0.014929** | val, T = 0.9773 | 16,441 | `calibration_v001.json` β `metrics.ece_before` / `.ece_after` | MEASURED β **worse** |
|
| 701 |
+
| System | end-to-end accuracy | β | β | β | β | **NOT RUN β none exists** |
|
| 702 |
|
| 703 |
+
---
|
| 704 |
+
|
| 705 |
+
## 9. Calibration β a measured negative result
|
| 706 |
|
| 707 |
+
Temperature scaling is **enabled** in the frozen configuration (`confidence.temperature_scaling: true`,
|
| 708 |
+
`confidence.calibration_file: calibration_v001.json`) and applied by
|
| 709 |
+
`evidence.confidence.TemperatureCalibration` as `sigmoid(logit(z)/T)` for a scalar `z` and
|
| 710 |
+
`softmax(logits/T)` for a distribution. Source: `artifacts/calibration_v001.json`.
|
| 711 |
|
| 712 |
+
| Field | Value | | Field | Value |
|
| 713 |
+
|---|---|---|---|---|
|
| 714 |
+
| `method` | `temperature_scaling` | | `temperature` | **0.9772731820958189** |
|
| 715 |
+
| `fitted_on` / `n_samples` | Val / **16,441** | | `n_classes` / `space` | 19 / `multiclass_logits` |
|
| 716 |
+
| `objective` | `mean_negative_log_likelihood` | | `optimizer` | `golden_section_on_log_temperature` (200 iters, `hit_bound: false`) |
|
| 717 |
+
| NLL before β after | 0.689741 β 0.689631 (Ξ 0.00011) | | **ECE before β after** | **0.013755 β 0.014929** |
|
| 718 |
+
| `ece_improvement` | **β0.001174** (negative β did **not** help) | | `n_bins` | 15 |
|
| 719 |
+
| scope | `change_vqa` only β "Other specialists emit their own raw scores and are unaffected." | | `type_mask_applied` | `false` |
|
| 720 |
+
| held-out splits excluded | `[Test, Test2]` | | | |
|
| 721 |
|
| 722 |
+
**The honest reading: the ECE got worse.** Temperature scaling reduced the NLL very slightly (0.00011)
|
| 723 |
+
but **increased** the expected calibration error from 0.013755 to 0.014929. It is retained **only
|
| 724 |
+
because it is part of the frozen configuration** β not because it helped. This is a measured negative
|
| 725 |
+
result and is reported as one (`docs/BENCHMARKS.md` Β§4.7, `docs/MODELS.md` Β§5).
|
| 726 |
|
| 727 |
+
**Two caveats on the number.** The artifact notes that "ECE is bin-count sensitive and is not an
|
| 728 |
+
aggregate score", and the `reliability_diagram` it carries is the **pre-scaling** curve (`ece 0.013755`),
|
| 729 |
+
labelled as such β the calibrated curve is **NOT plotted** (`docs/LIMITATIONS.md` Β§3.22).
|
|
|
|
|
|
|
|
|
|
|
|
|
| 730 |
|
| 731 |
+
---
|
| 732 |
|
| 733 |
+
## 10. Acceptance status
|
| 734 |
|
| 735 |
+
| Artifact | Metrics | Acceptance | Notes |
|
| 736 |
+
|---|---|---|---|
|
| 737 |
+
| `change` | VERIFIED | accepted (shipped) | the only `VERIFIED` headline |
|
| 738 |
+
| `grounding` | measured (2 protocols Γ 2 decode variants) | shipped | trained head is the production default; zero-shot is a labelled fallback (D-4) |
|
| 739 |
+
| `optical_sar` | measured | **ruling OPEN** | accuracy 0.931 always with macro-F1 0.434161 |
|
| 740 |
+
| `change_vqa` | measured (2 test sets) | **ruling OPEN** | test + test2 both reported |
|
| 741 |
+
| `router` | measured (val only) | shipped; **test NOT RUN** | 0.965116 is validation, ungated, n = 86 |
|
| 742 |
+
| **`vlm`** | usable (exact_match 0.963, F1 0.96432) | **ACCEPTANCE-REJECTED** | deployed path uses the unadapted model |
|
| 743 |
|
| 744 |
+
**`USABLE_VERIFIED` β `ACCEPTANCE-ACCEPTED`.** The VLM adapter works and is not promoted. The two
|
| 745 |
+
questions β *is this the artifact we trained, and does it work?* versus *did it clear the bar we
|
| 746 |
+
predeclared before looking?* β are kept separate on purpose (`docs/PHASE6_CLOSURE.md` Β§1).
|
| 747 |
+
|
| 748 |
+
---
|
| 749 |
+
|
| 750 |
+
## 11. Evaluation gaps (stated, not hidden)
|
| 751 |
+
|
| 752 |
+
| Gap | State |
|
| 753 |
+
|---|---|
|
| 754 |
+
| **System-level end-to-end benchmark** | **NOT RUN β none exists.** No end-to-end accuracy is claimed. |
|
| 755 |
+
| **Router test split** | **NOT RUN** (`test_split_touched: false`) |
|
| 756 |
+
| **Benchmark adapters** | **NOT RUN** |
|
| 757 |
+
| **End-to-end latency benchmark** | **NOT RUN** (per-specialist latency recorded only incidentally) |
|
| 758 |
+
| **Cross-dataset generalisation** | **NOT RUN** β each specialist is evaluated only on its own training-family split |
|
| 759 |
+
| **Human evaluation** | **NOT RUN** |
|
| 760 |
+
| **Robustness / adversarial evaluation** | **NOT RUN** |
|
| 761 |
+
| **Statistical significance for most metrics** | only the grounding 448-vs-224 decision has a paired test with a CI; other per-task numbers are point estimates |
|
| 762 |
+
| **Calibrated reliability curve** | **NOT plotted** |
|
| 763 |
+
| **BigEarthNet label semantics** | the local subset is **100 % single-label** vs the official 1β11 multi-label scheme, so its metrics are **not comparable** to published numbers |
|
| 764 |
+
|
| 765 |
+
---
|
| 766 |
+
|
| 767 |
+
## 12. Limitations
|
| 768 |
+
|
| 769 |
+
A condensed catalogue; the full version is [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Per-artifact
|
| 770 |
+
limitations are in Β§7; evaluation gaps in Β§11.
|
| 771 |
+
|
| 772 |
+
**Model quality.** Grounding absolute IoU is low (0.2838 / 0.2566) and **protocol-sensitive**.
|
| 773 |
+
Optical-SAR accuracy is carried by common classes β **0.931 with macro-F1 0.434161**. Change-VQA is weak
|
| 774 |
+
on rare classes (0.697626/0.378373 and 0.651469/0.372309). The live optical-SAR service returns a **bare
|
| 775 |
+
class index** (`class_18`), not a CLC label. The VLM adapter is **not accepted**. Calibration made ECE
|
| 776 |
+
**worse**.
|
| 777 |
+
|
| 778 |
+
**Router.** 0.965116 is **validation, ungated, n = 86**, corpus-limited; the test split was **NOT RUN**.
|
| 779 |
+
Known residuals: *"What is the new runway?"* reads `change`; *"How much built-up area was added?"*
|
| 780 |
+
under-triggers `vqa`; with one asset the console *reads* `change` while dispatch correctly falls back to
|
| 781 |
+
`change_vqa` (intentional, but visually surprising).
|
| 782 |
+
|
| 783 |
+
**Evaluation.** No end-to-end benchmark; no cross-dataset, human or robustness evaluation; most metrics
|
| 784 |
+
are point estimates without confidence intervals. The BigEarthNet local subset is **100 % single-label**,
|
| 785 |
+
so its metrics are **not comparable** to published multi-label numbers.
|
| 786 |
+
|
| 787 |
+
**Operational.** Transient tunnel gaps (B-07) β a request can hang or return 504; patch prepared but
|
| 788 |
+
**NOT deployed**. `OPEN`. `/api/health` `codespace_name` carries a trailing newline (B-02) β cosmetic.
|
| 789 |
+
`OPEN`. Cold start is tens of seconds; single-region, no HA; no database, auth or queue (stateless by
|
| 790 |
+
design).
|
| 791 |
+
|
| 792 |
+
**Packaging and licensing.** **No `LICENSE` file exists** in the source repository. `OPEN`. The six
|
| 793 |
+
artifacts require their pinned backbones, which are **not** redistributed.
|
| 794 |
+
|
| 795 |
+
**Documentation.** `docs/FINAL_DELIVERY_REPORT.md` Β§6 is stale (it lists the bundled EO change pair as
|
| 796 |
+
DEGRADED and B-01 as BLOCKED; both were resolved on 2026-09-25). The original master plan describes a
|
| 797 |
+
**superseded deployment** (Gradio GUI + HF Space + ZeroGPU + Railway); the shipped system is a static
|
| 798 |
+
frontend + Render + Codespace tunnel serving JSON.
|
| 799 |
+
|
| 800 |
+
**Explicit non-claims.** No state-of-the-art claim; no production-readiness claim for model quality; no
|
| 801 |
+
claim that the trained heads generalise beyond their training-family splits; no claim that calibration
|
| 802 |
+
improves confidence; no claim that the VLM adapter is accepted; no end-to-end accuracy claim; no
|
| 803 |
+
robustness claim; no geolocation-accuracy claim; not a safety-, legal- or life-critical tool.
|
| 804 |
+
|
| 805 |
+
---
|
| 806 |
+
|
| 807 |
+
## 13. Training summary
|
| 808 |
+
|
| 809 |
+
All six artifacts are small modules on frozen backbones, trained with **seed 42** and recording the
|
| 810 |
+
frozen config hash **`78f1e3700da15aa1`**.
|
| 811 |
+
|
| 812 |
+
| Artifact | Where it trained | Precision | Notable settings |
|
| 813 |
+
|---|---|---|---|
|
| 814 |
+
| `change` | GPU (eval artifact: `device cuda`, torch 2.10.0+cu128) | β | STANet-style; ResNet-18 pretrained; PAM; bce 0.5 + dice 0.5 |
|
| 815 |
+
| `change_vqa` | **external GPU (Kaggle)** | fp16 | epoch 8 selected on val answer accuracy 0.700018; early stopping |
|
| 816 |
+
| `optical_sar` | in-project | β | arm A retained (A 0.837100 vs B 0.839100, floor 0.0285); 10 runs Γ 2 arms Γ 5 seeds |
|
| 817 |
+
| `grounding` | CPU | β | 20 epochs; grid 7Γ7; resolution frozen at 224 by pre-registered test |
|
| 818 |
+
| `router` | **CPU** | β | frozen encoder, cached embeddings; 20 epochs / 4,096 vectors in 0.28 s |
|
| 819 |
+
| `vlm` | **external GPU (Kaggle, T4)** | fp16 | LoRA r=16 Ξ±=32; 224 text-projection modules; vision tower untouched |
|
| 820 |
+
|
| 821 |
+
**Precision.** `training.precision: fp16` because the target GPU (T4) is compute capability 7.5 β bf16 is
|
| 822 |
+
unavailable there (finding C-6). The loader validates the value is one of `fp16|bf16|fp32`.
|
| 823 |
+
|
| 824 |
+
**Provenance discipline.** `change_vqa` was promoted from a Kaggle export with 93/0/0 verification
|
| 825 |
+
checks and a byte-identical source copy. The VLM adapter's Phase 6 closure was reached **without
|
| 826 |
+
retraining or modifying** the adapter β the promoted weights are the end-of-training top-level save, and
|
| 827 |
+
the closure record is generated from the evidence rather than restated. Full procedures:
|
| 828 |
+
[`docs/TRAINING.md`](docs/TRAINING.md); dataset provenance: [`docs/DATASETS.md`](docs/DATASETS.md).
|
| 829 |
+
|
| 830 |
+
---
|
| 831 |
+
|
| 832 |
+
## 14. Provenance and verification
|
| 833 |
|
| 834 |
| Item | Location |
|
| 835 |
|---|---|
|
| 836 |
+
| Byte-exact manifest (generated from disk) | [`models/manifest.json`](models/manifest.json) |
|
| 837 |
+
| Checksums | [`models/checksums.sha256`](models/checksums.sha256) |
|
| 838 |
+
| Metric verification tool / output | [`tools/verify_readme_metrics.py`](tools/verify_readme_metrics.py) Β· [`tools/readme_metrics_report.txt`](tools/readme_metrics_report.txt) |
|
| 839 |
+
| HF release verification (re-downloaded, 6/6 MATCH) | [`HF_RELEASE_VERIFICATION.md`](HF_RELEASE_VERIFICATION.md) |
|
| 840 |
+
| Release manifest (every file, size + sha256) | [`RELEASE_MANIFEST.md`](RELEASE_MANIFEST.md) |
|
| 841 |
+
| Full documentation / repository front page | [`docs/`](docs/) Β· [`README.md`](README.md) |
|
| 842 |
+
|
| 843 |
+
**Verification chain.** (1) `models/manifest.json` and `models/checksums.sha256` are **generated by
|
| 844 |
+
reading the files** (`release/tools/generate_model_manifest.py`); no byte count or hash is typed by hand.
|
| 845 |
+
(2) `tools/verify_readme_metrics.py` walks every quoted metric to its source artifact; result
|
| 846 |
+
**`ALL CLAIMS VERIFIED`** (20/20), with status assertions (VLM `ACCEPTANCE-REJECTED`, router
|
| 847 |
+
`corpus_limited` n = 86, calibration `ece_improvement` negative) confirmed. (3)
|
| 848 |
+
`release/tools/hf_verify.py` re-downloads each artifact over direct HTTPS and hashes the received bytes;
|
| 849 |
+
**6/6 MATCH, 0 failed**. (4) Two artifact digests agree with values recorded independently at
|
| 850 |
+
promotion/fit time (Β§3).
|
| 851 |
+
|
| 852 |
+
**Hugging Face release.** `thundercode/SatQuery` (public), HEAD `bf2779e18fcaa7476b93a48a978f08c108dfdfb7`,
|
| 853 |
+
`lastModified 2026-09-25T21:46:52Z`, 42 files on the Hub. No secret was uploaded; the token used is not
|
| 854 |
+
written into any released file (`HF_RELEASE_VERIFICATION.md` Β§8).
|
| 855 |
+
|
| 856 |
+
**GitHub release.** The curated public repository target is `Anish-lab-blip/SatQuery-AI`; the release
|
| 857 |
+
tree is staged and its links verified. At the time of `release/RELEASE_EXECUTION_CHECKLIST.md` Phase 5,
|
| 858 |
+
the push was **BLOCKED** because the fine-grained token was read-only for repository contents
|
| 859 |
+
(`403 Resource not accessible by personal access token`) β an owner action, not a defect in this release.
|
| 860 |
+
Where this card and a live repository disagree, treat the live repository as authoritative for
|
| 861 |
+
publication state and this card as authoritative for artifact identity.
|
| 862 |
+
|
| 863 |
+
---
|
| 864 |
|
| 865 |
+
## 15. Licence
|
| 866 |
|
| 867 |
+
The project ships **no licence file**; a licence must be selected by the owner before public release of
|
| 868 |
+
the *code* (`docs/LIMITATIONS.md` Β§5, `RELEASE_MANIFEST.md`). This is an **OPEN** item. The Hugging Face
|
| 869 |
+
card declares `license: other` because the correct licence has not yet been chosen.
|
| 870 |
|
| 871 |
+
**Model weights carry the terms of their backbone licences.** The six artifacts are small modules, but
|
| 872 |
+
they depend on and are intended to be used with: `sentence-transformers/all-MiniLM-L6-v2`;
|
| 873 |
+
`HuggingFaceTB/SmolVLM-500M-Instruct`; `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`);
|
| 874 |
+
`antofuller/CROMA` (`CROMA_base.pt`); and torchvision ResNet-18 (`IMAGENET1K_V1`). **Backbones are not
|
| 875 |
+
redistributed here.** Consult each backbone's Hugging Face page for the authoritative licence β the
|
| 876 |
+
licence labels above are recorded for convenience and should be verified at the source before reuse.
|
| 877 |
+
|
| 878 |
+
---
|
| 879 |
+
|
| 880 |
+
## 16. Citation
|
| 881 |
|
| 882 |
If you use this work, cite the project repository:
|
| 883 |
|
|
|
|
| 889 |
note = {Public release: https://github.com/Anish-lab-blip/SatQuery-AI}
|
| 890 |
}
|
| 891 |
```
|
| 892 |
+
|
| 893 |
+
**Model card version.** This card documents release **1.0.0** (2026-09-25), frozen config hash
|
| 894 |
+
`78f1e3700da15aa1`. The changelog is [`docs/CHANGELOG.md`](docs/CHANGELOG.md).
|