thundercode commited on
Commit
d123a4c
Β·
verified Β·
1 Parent(s): 258267c

release: add MODEL_CARD.md

Browse files
Files changed (1) hide show
  1. MODEL_CARD.md +816 -86
MODEL_CARD.md CHANGED
@@ -20,7 +20,7 @@ config_hash: 78f1e3700da15aa1
20
  # Model Card β€” SatQuery AI
21
 
22
  SatQuery AI answers natural-language questions about satellite imagery using a **router + specialists**
23
- design. This card covers the **six trained artifacts** released by the project. It is deliberately
24
  explicit about what is measured, what is not, and what was rejected.
25
 
26
  > **The six trained artifacts are small modules on top of frozen, publicly-pinned backbones. No
@@ -29,128 +29,855 @@ explicit about what is measured, what is not, and what was rejected.
29
 
30
  Machine-readable identities (byte counts and sha256) are in
31
  [`models/manifest.json`](models/manifest.json) and [`models/checksums.sha256`](models/checksums.sha256),
32
- **generated by reading the files**. Where this card and the generated manifest disagree, the manifest
33
- wins.
 
 
 
 
 
 
34
 
35
  ---
36
 
37
- ## 1. Artifacts in this release
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
 
39
- | # | Task | Kind | File | Bytes | sha256 (first 16) |
40
- |---|---|---|---|---|---|
41
- | 1 | `change` | trained head | `head.pt` | 63,231,009 | `c5ef31277b67aa01` |
42
- | 2 | `change_vqa` | trained head | `head.pt` | 5,822,809 | `cfae5e43b97ca930` |
43
- | 3 | `optical_sar` | trained head | `head.pt` | 14,427,457 | `785815729a3a39fc` |
44
- | 4 | `grounding` | trained head | `head.pt` | 12,639,041 | `93432f7034be91a8` |
45
- | 5 | `router` | trained adapter | `adapter.pt` | 211,961 | `8527c3ed28a293e1` |
46
- | 6 | `vlm` | LoRA adapter | `adapter_model.safetensors` | 34,798,048 | `07c76a75fa046248` |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
 
48
- Two of these hashes (`change_vqa`, `vlm`) **agree exactly** with hashes recorded independently at
49
- promotion time β€” an external cross-check, not a self-consistency claim.
 
 
 
50
 
51
- ## 2. Backbone dependencies (frozen, pinned by revision)
 
52
 
53
- | Role | Repository | Revision |
 
 
54
  |---|---|---|
55
- | Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` |
56
- | VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` |
57
- | Grounding | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`) | `bf1d8a3ccf2d` |
58
- | Optical-SAR | `antofuller/CROMA` (`CROMA_base.pt`) | `0dd28e3d633b` |
59
- | Change | STANet-style (ResNet-18 + PAM) | trained in-project |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
60
 
61
- ## 3. Intended use
 
 
62
 
63
- - **Research and demonstration** of a modular, CPU-first remote-sensing QA system.
64
- - **Routing and dispatch** of natural-language queries to the appropriate specialist.
65
- - **Reproducible evaluation** of each specialist on its own documented split.
 
 
 
 
 
66
 
67
- ## 4. Out-of-scope use
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
 
69
  - **Safety-, legal- or life-critical decisions.** No accuracy, calibration or robustness guarantee is
70
- offered for any high-stakes use.
 
71
  - **Operational geospatial production** without independent validation.
72
- - **Any use of the VLM adapter as a production model** β€” it is acceptance-rejected (Β§6).
73
- - **Treating per-specialist metrics as system-level accuracy.** No end-to-end benchmark exists (Β§7).
 
 
 
 
 
 
74
 
75
- ## 5. Measured performance
76
 
77
- | Task | Metric | Value | Split / protocol |
78
- |---|---|---|---|
79
- | change | pooled IoU / macro IoU / pooled F1 | **0.8122 / 0.8457 / 0.8964** | LEVIR-CD-256 test, n = 2048 |
80
- | grounding | mean best IoU / recall@0.5 | **0.2838 / 0.2198** canonical; **0.2566 / 0.1938** matched6 | VRSBench, n = 16159 |
81
- | grounding | head-argmax / zero-shot baseline IoU | **0.1215 / 0.0972** | canonical |
82
- | optical_sar | accuracy / macro F1 | **0.931 / 0.434161** | held-out test, n = 4000, 19 classes |
83
- | change_vqa | accuracy / macro F1 | **0.697626 / 0.378373** (test); **0.651469 / 0.372309** (test2) | two test sets |
84
- | vlm | exact_match / F1 | **0.963 / 0.96432** | frozen 1000-question subset |
85
- | router | overall **ungated** accuracy | **0.965116** | val, n = 86 β€” **TEST NOT RUN** |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
 
87
- **Every value is checked against its artifact** by `tools/verify_readme_metrics.py`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
 
89
- ### 5.1 Calibration β€” reported as a negative result
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
90
 
91
- Temperature scaling is enabled (T = 0.9772732, fit on val n = 16,441). **ECE worsened**:
92
- 0.013755 β†’ **0.014929**. It is retained because it is part of the frozen configuration, not because it
93
- helped.
94
 
95
- ## 6. Acceptance status
 
 
 
 
96
 
97
- | Artifact | Metrics | Acceptance |
 
 
98
  |---|---|---|
99
- | change | VERIFIED | accepted (shipped) |
100
- | grounding | measured (2 protocols) | shipped |
101
- | optical_sar | measured | **ruling OPEN** |
102
- | change_vqa | measured (2 test sets) | **ruling OPEN** |
103
- | router | measured (val only) | shipped; test NOT RUN |
104
- | **vlm** | usable (exact_match 0.963) | **ACCEPTANCE-REJECTED** |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
 
106
- **`USABLE_VERIFIED` β‰  `ACCEPTANCE-ACCEPTED`.** The VLM adapter works and is not promoted; the deployed
107
- caption/VQA path uses the **unadapted** model.
 
108
 
109
- ## 7. Evaluation gaps (stated, not hidden)
 
 
 
110
 
111
- - **No system-level end-to-end benchmark exists.** None is claimed.
112
- - **Router test split: NOT RUN.**
113
- - **Benchmark adapters: NOT RUN.**
114
- - **Cross-dataset generalisation: NOT RUN.**
115
- - **Human and robustness evaluation: NOT RUN.**
 
 
 
 
116
 
117
- ## 8. Limitations
 
 
 
118
 
119
- - Grounding absolute IoU is low (0.28) and protocol-sensitive.
120
- - Optical-SAR accuracy is carried by common classes (macro-F1 0.434161).
121
- - The BigEarthNet local subset is **100 % single-label** vs the official 1–11 multi-label scheme, so
122
- its metrics are **not comparable** to published numbers.
123
- - The optical-SAR service returns a bare class index, not a label.
124
- - Known router residuals exist (e.g. *"What is the new runway?"* reads `change`).
125
- - No `LICENSE` file exists in the source repository.
126
 
127
- See [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md) for the full catalogue.
128
 
129
- ## 9. Training summary
130
 
131
- Small modules on frozen backbones; seed 42; every artifact records the frozen config hash
132
- `78f1e3700da15aa1`. Router and change/grounding/fusion heads train on CPU; the VLM LoRA adapter and
133
- the change-VQA head were trained on **external GPUs** (the latter via a documented Kaggle run). Full
134
- detail in [`docs/TRAINING.md`](docs/TRAINING.md).
 
 
 
 
135
 
136
- ## 10. Provenance and verification
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137
 
138
  | Item | Location |
139
  |---|---|
140
- | Byte-exact manifest | `models/manifest.json` |
141
- | Checksums | `models/checksums.sha256` |
142
- | Metric verification tool | `tools/verify_readme_metrics.py` |
143
- | Metric verification output | `tools/readme_metrics_report.txt` |
144
- | Full documentation | `docs/` |
145
- | Release manifest | `RELEASE_MANIFEST.md` |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
146
 
147
- ## 11. Licence
148
 
149
- The project ships **no licence file**; a licence must be selected by the owner before public release
150
- of the *code*. **Model weights carry the terms of their backbone licences** β€” consult each backbone's
151
- Hugging Face page. Backbones are not redistributed here.
152
 
153
- ## 12. Citation
 
 
 
 
 
 
 
 
 
154
 
155
  If you use this work, cite the project repository:
156
 
@@ -162,3 +889,6 @@ If you use this work, cite the project repository:
162
  note = {Public release: https://github.com/Anish-lab-blip/SatQuery-AI}
163
  }
164
  ```
 
 
 
 
20
  # Model Card β€” SatQuery AI
21
 
22
  SatQuery AI answers natural-language questions about satellite imagery using a **router + specialists**
23
+ design. This card documents the **six trained artifacts** released by the project. It is deliberately
24
  explicit about what is measured, what is not, and what was rejected.
25
 
26
  > **The six trained artifacts are small modules on top of frozen, publicly-pinned backbones. No
 
29
 
30
  Machine-readable identities (byte counts and sha256) are in
31
  [`models/manifest.json`](models/manifest.json) and [`models/checksums.sha256`](models/checksums.sha256),
32
+ **generated by reading the files** (`release/tools/generate_model_manifest.py`). Where this card and the
33
+ generated manifest disagree, the manifest wins β€” it is computed from disk, this card is written by hand.
34
+
35
+ **Companion documents** (same depth, same no-fabrication rule): [`docs/MODELS.md`](docs/MODELS.md) Β·
36
+ [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) Β· [`docs/TRAINING.md`](docs/TRAINING.md) Β·
37
+ [`docs/EVALUATION.md`](docs/EVALUATION.md) Β· [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md) Β·
38
+ [`docs/RESEARCH_NOTES.md`](docs/RESEARCH_NOTES.md) Β·
39
+ [`docs/architecture/05-specialists.md`](docs/architecture/05-specialists.md).
40
 
41
  ---
42
 
43
+ ## Table of contents
44
+
45
+ 1. [How to read this card](#1-how-to-read-this-card)
46
+ 2. [Overview](#2-overview)
47
+ 3. [The six artifacts in this release](#3-the-six-artifacts-in-this-release)
48
+ 4. [Backbone dependencies β€” frozen, pinned by revision](#4-backbone-dependencies--frozen-pinned-by-revision)
49
+ 5. [Intended use](#5-intended-use)
50
+ 6. [Out-of-scope use](#6-out-of-scope-use)
51
+ 7. [Per-artifact reference](#7-per-artifact-reference)
52
+ 8. [Full measured-performance table](#8-full-measured-performance-table)
53
+ 9. [Calibration β€” a measured negative result](#9-calibration--a-measured-negative-result)
54
+ 10. [Acceptance status](#10-acceptance-status)
55
+ 11. [Evaluation gaps](#11-evaluation-gaps)
56
+ 12. [Limitations](#12-limitations)
57
+ 13. [Training summary](#13-training-summary)
58
+ 14. [Provenance and verification](#14-provenance-and-verification)
59
+ 15. [Licence](#15-licence)
60
+ 16. [Citation](#16-citation)
61
 
62
+ ---
63
+
64
+ ## 1. How to read this card
65
+
66
+ **The single most important rule in this document: do not fabricate.** Every byte count, sha256,
67
+ hyperparameter and metric below comes from a file that was read, and each one names its source. Where a
68
+ fact is not established, this card writes `UNKNOWN β€” not established from the available evidence`
69
+ rather than estimating.
70
+
71
+ **Status vocabulary.** Every substantive claim carries one of: `IMPLEMENTED` Β· `VERIFIED` Β· `MEASURED` Β·
72
+ `ATTEMPTED` Β· `NOT RUN` Β· `BLOCKED` Β· `DEFERRED` Β· `REJECTED` Β· `OPEN` Β· `RESOLVED` Β· `CLOSED`.
73
+
74
+ **The facts most easily stated wrongly, and therefore stated repeatedly:**
75
+
76
+ | Fact | Correct statement |
77
+ |---|---|
78
+ | Grounding | measured under **two protocols** (canonical 0.2838 / 0.2198; matched6 0.2566 / 0.1938) **and two decode variants** (head_argmax 0.1215; zero-shot 0.0972). Never quote one alone. |
79
+ | Calibration | ECE went **0.013755 β†’ 0.014929 β€” worse**. Retained only because it is in the frozen config. |
80
+ | VLM adapter | metrics **usable** (exact_match 0.963) but status **ACCEPTANCE-REJECTED**. USABLE β‰  ACCEPTED. |
81
+ | Optical-SAR | accuracy **0.931** with macro-F1 **0.434161**; ruling **OPEN**. Never accuracy without macro-F1. |
82
+ | Change-VQA | **two** test sets: test 0.697626/0.378373 and test2 0.651469/0.372309; ruling **OPEN**. |
83
+ | Router | **0.965116 is validation, ungated, n = 86**; the **test split was NOT RUN**. |
84
+ | End-to-end benchmark | **does not exist**; no system-level accuracy is claimed. |
85
+ | Change | pooled IoU 0.8122 / macro IoU 0.8457 / pooled F1 0.8964 β€” the **only** `VERIFIED` headline. |
86
+
87
+ ---
88
+
89
+ ## 2. Overview
90
 
91
+ SatQuery AI is a **router-and-specialists** system: a frozen sentence encoder plus a small trained
92
+ adapter classify a query into one of six tasks; a deterministic planner dispatches it to the
93
+ appropriate specialist; each specialist returns a structured `ResultEnvelope` carrying evidence and a
94
+ confidence value. The design is **CPU-first** and **frozen-backbone** β€” small modules are trained on top
95
+ of pretrained encoders, and no encoder is fine-tuned end to end.
96
 
97
+ **The six tasks** (`configs/base.yaml` β†’ `router.tasks`):
98
+ `vqa` Β· `caption` Β· `grounding` Β· `change` Β· `optical_sar` Β· `unsupported`.
99
 
100
+ **The six trained artifacts** (details in Β§3):
101
+
102
+ | Task | What it is | Size |
103
  |---|---|---|
104
+ | `change` | STANet-style Siamese change detector (ResNet-18 + PAM) | 63,231,009 B |
105
+ | `change_vqa` | two-stage change-reasoning head (`change_vqa_head_v1`) | 5,822,809 B |
106
+ | `optical_sar` | CROMA-base fusion head (2318 β†’ 512 β†’ 19) | 14,427,457 B |
107
+ | `grounding` | RemoteCLIP grounding head (feature 2048 β†’ hidden 512) | 12,639,041 B |
108
+ | `router` | five-head intent adapter over frozen MiniLM | 211,961 B |
109
+ | `vlm` | PEFT LoRA adapter on SmolVLM-500M text projections | 34,798,048 B |
110
+
111
+ Total released weight payload: **131,130,325 bytes (~125 MiB)** (`HF_RELEASE_VERIFICATION.md` Β§4).
112
+
113
+ **What is not trained here.** MiniLM, SmolVLM-500M, RemoteCLIP ViT-B/32 and CROMA-base are frozen and
114
+ not redistributed. The one nuance is the change detector's ResNet-18, which is *loaded pretrained*
115
+ (`pretrained_used: true`) and *trained in-project* as part of the change head β€” so its weights are part
116
+ of the released `change/head.pt`, not a separately-distributed backbone
117
+ (`artifacts/change/eval_test/eval_result.json` β†’ `checkpoint_embedded_config`).
118
+
119
+ **No end-to-end accuracy is claimed anywhere.** The router β†’ specialist β†’ envelope pipeline has never
120
+ been scored end to end. What exists is per-specialist metrics on their own training-family splits (Β§8)
121
+ and a behavioural live-validation record that proves the pipeline *runs and routes* β€” 3 passes Γ— 8
122
+ cases, 8/8 each, **24 live runs, 0 mock nodes**, trace fill **94.4444 %** (`docs/BENCHMARKS.md` Β§5,
123
+ `docs/RESEARCH_NOTES.md` Β§3.3).
124
+
125
+ ---
126
+
127
+ ## 3. The six artifacts in this release
128
 
129
+ Reproduced from [`models/manifest.json`](models/manifest.json) β†’ `artifacts[*]`, cross-checked against
130
+ [`models/checksums.sha256`](models/checksums.sha256). Every artifact carries `status: "PRESENT"` and
131
+ `config_hash: "78f1e3700da15aa1"`.
132
 
133
+ | # | `id` | Task | Kind | File (HF path) | Bytes | sha256 (full) |
134
+ |---|---|---|---|---|---|---|
135
+ | 1 | `change_head` | `change` | trained head | `change/head.pt` | 63,231,009 | `c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa` |
136
+ | 2 | `change_vqa_head` | `change_vqa` | trained head | `change_vqa/head.pt` | 5,822,809 | `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` |
137
+ | 3 | `optical_sar_fusion_head` | `optical_sar` | trained head | `optical_sar/head.pt` | 14,427,457 | `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab` |
138
+ | 4 | `grounding_head` | `grounding` | trained head | `grounding/head.pt` | 12,639,041 | `93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb` |
139
+ | 5 | `router_adapter` | `router` | trained adapter | `router/adapter.pt` | 211,961 | `8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` |
140
+ | 6 | `vlm_lora_adapter` | `vlm` | LoRA adapter | `vlm/adapter_model.safetensors` | 34,798,048 | `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` |
141
 
142
+ The manifest also records each artifact's **original repository path** and its **source metric
143
+ artifact**:
144
+
145
+ | # | `id` | `path` (source repo) | `source_metric_artifact` |
146
+ |---|---|---|---|
147
+ | 1 | `change_head` | `artifacts/change/levir_change_v001/head.pt` | `artifacts/change/eval_test/eval_result.json` |
148
+ | 2 | `change_vqa_head` | `artifacts/change_vqa/run/head.pt` | `artifacts/change_vqa/run/PROMOTION.json` |
149
+ | 3 | `optical_sar_fusion_head` | `artifacts/optical_sar/fusion_head_production_v001/head.pt` | `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json` |
150
+ | 4 | `grounding_head` | `artifacts/grounding/remoteclip_grounding_v001/head.pt` | `artifacts/grounding/remoteclip_grounding_v001/eval_result_canonical.json` |
151
+ | 5 | `router_adapter` | `artifacts/router/router_adapter_v001/adapter.pt` | `artifacts/router/threshold_sweep_val.json` |
152
+ | 6 | `vlm_lora_adapter` | `.scratch/phase6_real_adapter/phase6_adapter/adapter_model.safetensors` | `artifacts/vlm/phase6_closure.json` |
153
+
154
+ **`kind` semantics.** `trained_head` = a module trained in-project on a frozen encoder, loaded via
155
+ `torch.load` of a `state_dict` (or the module's own loader). `trained_adapter` = a small classifier over
156
+ a frozen sentence encoder's cached embeddings, loaded via `IntentAdapter.from_config_dict` +
157
+ `load_state_dict` (`router/adapter.py`). `lora_adapter` = a PEFT LoRA delta attached at load time via
158
+ `peft.PeftModel.from_pretrained(model, dir)` (`specialists/vqa/model.py`).
159
+
160
+ **Two independent cross-checks (not self-consistency).** The manifest is generated by hashing the files
161
+ on disk; for two artifacts the computed digest can be compared against a value recorded independently,
162
+ at a different time, by a different process:
163
+
164
+ - **`change_vqa` = `cfae5e43…d63a82a`.** Equals `artifacts/change_vqa/run/PROMOTION.json` β†’
165
+ `artifact.sha256`, the digest recorded in the Kaggle run record *before* promotion
166
+ (`source.checkpoint_sha256_in_run_record`), and `artifacts/calibration_v001.json` β†’
167
+ `provenance.checkpoint_sha256` (recorded when the temperature was fitted β€” a separate step).
168
+ `PROMOTION.json` β†’ `source.hash_agrees_across` records the digest agreeing across
169
+ `model_metadata.json`, `run_record.json` and `hashes.json`, with `byte_identical_to_source: true` and
170
+ `artifact.weights_modified: false`.
171
+ - **`vlm` = `07c76a75…a5adf5e`.** Equals `artifacts/vlm/phase6_closure.json` β†’
172
+ `why_usable_verified.adapter_provenance` β†’ `adapter_verification.json` β†’ `weights_file_sha256`, and the
173
+ adapter's own `ARTIFACT_SHA256SUMS.json`, against which the 14-file directory was verified
174
+ (`manifest_check.clean: true`, 14/14 present).
175
+
176
+ A third, independent re-download check: `release/tools/hf_verify.py` re-downloads each artifact over
177
+ direct HTTPS and hashes the received bytes β€” **6/6 MATCH, 0 failed**
178
+ ([`HF_RELEASE_VERIFICATION.md`](HF_RELEASE_VERIFICATION.md) Β§5).
179
+
180
+ **On `parameters: null`.** Four of six artifacts record `parameters: null` deliberately β€” the generator
181
+ does not open checkpoints (that would make generation depend on the model code and torch). Counts
182
+ measured elsewhere appear in Β§7 with their source; where a count is not established this card writes
183
+ `UNKNOWN β€” not established from the available evidence`. Training checkpoints are **not** released
184
+ artifacts: the VLM adapter's `checkpoint-1500/`/`checkpoint-2000/` are provenance only, and the promoted
185
+ adapter is the top-level end-of-training save, not `checkpoint-2000` (Β§7.6).
186
+
187
+ ---
188
+
189
+ ## 4. Backbone dependencies β€” frozen, pinned by revision
190
+
191
+ Backbones are resolved from the Hugging Face Hub on first use, **pinned by revision** β€” a moving `main`
192
+ would make every benchmark number unreproducible.
193
+
194
+ | Role | Repository | Revision | Size | Measured identity | Notes |
195
+ |---|---|---|---|---|---|
196
+ | Router encoder | `sentence-transformers/all-MiniLM-L6-v2` | `1110a243fdf4` | 90.9 MB | 22,713,216 params, 384-dim | tokenizer ceiling **256**; truncation **128** |
197
+ | VLM | `HuggingFaceTB/SmolVLM-500M-Instruct` | `a7da5b986cb5` | ~1015 MB safetensors | 516,165,824 params (base) | processor `longest_edge` must be pinned (F5-2) |
198
+ | Grounding | `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`) | `bf1d8a3ccf2d` | 605.2 MB | 151,277,313 params; width **768**, projected **512** | patch 32; 7Γ—7 tokens at 224 |
199
+ | Optical-SAR | `antofuller/CROMA` (`CROMA_base.pt`) | `0dd28e3d633b` | 777.6 MB (777,563,846 B) | 194,365,440 params; `encoder_dim` 768 | resolution 120; asymmetric `s1_depth=6`, `s2_depth=12` |
200
+ | Change encoder | β€” (torchvision) | β€” | β€” | ResNet-18, `IMAGENET1K_V1` | `pretrained_used: true` in the artifact |
201
+
202
+ All pins are declared in `configs/base.yaml` under the `router:`, `vlm:`, `grounding:` and `croma:`
203
+ blocks and are validated at load time by `core/config.py`. Backbone licence terms are each repository's
204
+ own β€” see Β§15. **Backbones are not redistributed here.**
205
+
206
+ ---
207
+
208
+ ## 5. Intended use
209
+
210
+ - **Research and demonstration** of a modular, CPU-first remote-sensing question-answering system.
211
+ - **Routing and dispatch** of natural-language queries to the appropriate specialist, using the
212
+ `router` adapter over frozen MiniLM embeddings.
213
+ - **Reproducible evaluation** of each specialist on its own documented split, using the released
214
+ artifacts and the frozen config hash `78f1e3700da15aa1`.
215
+ - **Teaching and ablation**: the six artifacts are small and individually inspectable; the frozen-backbone
216
+ design makes each head a self-contained experiment.
217
+
218
+ The artifacts are intended to be used **with their pinned backbones** (Β§4), which the consumer must
219
+ fetch separately.
220
+
221
+ ---
222
+
223
+ ## 6. Out-of-scope use
224
 
225
  - **Safety-, legal- or life-critical decisions.** No accuracy, calibration or robustness guarantee is
226
+ offered. Grounding boxes are **image-relative, not geodetic** β€” no geolocation accuracy
227
+ (`docs/LIMITATIONS.md` Β§7).
228
  - **Operational geospatial production** without independent validation.
229
+ - **Any use of the VLM adapter as a production model** β€” it is `ACCEPTANCE-REJECTED` (Β§10); the deployed
230
+ caption/VQA path uses the **unadapted** model.
231
+ - **Treating per-specialist metrics as system-level accuracy.** No end-to-end benchmark exists (Β§11).
232
+ - **Any claim that these artifacts generalise beyond their training-family test splits** β€” cross-dataset
233
+ generalisation is `NOT RUN`.
234
+ - **Redistribution of the backbones.** This release contains no backbone weights.
235
+
236
+ ---
237
 
238
+ ## 7. Per-artifact reference
239
 
240
+ Each subsection gives architecture, hyperparameters (from `configs/base.yaml` unless noted), training
241
+ data, evaluation protocol, measured numbers (with source artifact and key path), acceptance status, and
242
+ limitations.
243
+
244
+ ### 7.0 Enforced configuration invariants, with arithmetic
245
+
246
+ The frozen registry is `configs/base.yaml`; the project rule is **"no magic numbers anywhere in Python;
247
+ everything tunable lives here"**, and `core/config.py` loads, validates and hashes every value. Several
248
+ values are *enforced* β€” a mismatch is a load-time error, not a comment:
249
+
250
+ | Invariant | Arithmetic / rule | Why it is enforced |
251
+ |---|---|---|
252
+ | Fusion input width | `3 Γ— 768 + 12 + 2 = 2318` | `core/config.py` recomputes it and `fusion_head.py` recomputes it **again**, refusing to build on mismatch β€” a config edit cannot silently reshape the first `Linear` |
253
+ | Grounding head feature width | `4 Γ— 512 = 2048` | `core/config.py` rejects any other value and `specialists/grounding/remoteclip.py` asserts it against the real model β€” a mismatch is a **silent** shape error otherwise |
254
+ | CROMA resolution | `image_resolution % 8 == 0`; native `120` β†’ `225` patches | required by CROMA (finding C-7) |
255
+ | Router truncation | `max_length ≀ 256` | MiniLM tokenizer ceiling; truncating above it is a silent no-op (F4-1) |
256
+ | VLM processor | `processor_longest_edge ≀ image.tile_size` | otherwise the processor upscales and splits a tile ~17Γ— (F5-2) |
257
+ | Change tile | `256`; `tile_overlap: 0` | STANet-style detector; LEVIR-CD-256 |
258
+ | Frozen config hash | `Config.hash = sha256(base.yaml)[:16] = 78f1e3700da15aa1` | every artifact records it; a config edit detaches the numbers from their configuration |
259
+
260
+ The frozen hash is verified untouched by `test_the_frozen_config_hash_has_not_moved`
261
+ (`docs/OWNER_DECISIONS_2026-09-23.md`, cross-cutting rule 4). New defaults live in **code**, not in the
262
+ registry β€” which is why the grounding head's default path is `DEFAULT_HEAD_PATH` in code rather than a
263
+ `base.yaml` key (owner decision D-4).
264
+
265
+ ---
266
+
267
+ ### 7.1 `change` β€” STANet-style Siamese change detector
268
+
269
+ **Kind:** trained head Β· **File:** `change/head.pt` Β· **Bytes:** 63,231,009 Β· **sha256:**
270
+ `c5ef31277b67aa01a593aec0eac503eeaccc6d674349fda20ca44c9cc6f8e9fa`
271
+
272
+ **Architecture.** A STANet-style Siamese detector: a shared ResNet-18 encoder (`SharedResNetEncoder`), a
273
+ `DifferenceFusion` module per stage, **PAM** spatial self-attention (`SpatialAttention`, `sa_mode: PAM`;
274
+ BAM is the alternative), a three-stage decoder (`dec3 β†’ dec2 β†’ dec1`) with a final upsample and a 1Γ—1
275
+ convolution head to one change logit. Source: `specialists/change/stanet.py`. Manifest architecture
276
+ string: `"STANet-style Siamese change detector (ResNet-18 + PAM)"`.
277
+
278
+ **Hyperparameters** (`base.yaml` β†’ `change:`; artifact `checkpoint_embedded_config`):
279
+
280
+ | Parameter | Value | | Parameter | Value |
281
+ |---|---|---|---|---|
282
+ | `tile_size` / `tile_overlap` | 256 / 0 | | `threshold` | 0.50 |
283
+ | `min_component_pixels` | 32 | | `encoder` | `resnet18` |
284
+ | `encoder_channels` | `[64,128,256,512]` | | `width` | 128 |
285
+ | `sa_mode` | `PAM` | | `pretrained` | `true` |
286
+ | `frozen_encoder` | `false` | | `attention_budget_bytes` | 268,435,456 |
287
+ | `learning_rate` / `batch_size` | 0.001 / 8 | | `bce_weight` / `dice_weight` | 0.5 / 0.5 |
288
+
289
+ **Training data.** LEVIR-CD-256, split `train 7,120 / val 1,024 / test 2,048` (`base.yaml` β†’
290
+ `change.levir_split`), matching the published LEVIR-CD counts exactly
291
+ (`docs/OWNER_DECISIONS_2026-09-23.md` D-11). Trained on GPU (eval artifact: `device: cuda`,
292
+ torch 2.10.0+cu128, python 3.12.13).
293
+
294
+ **Evaluation protocol.** Held-out `test` split, n = 2048, threshold 0.50, tile 256, no overlap. Source:
295
+ `artifacts/change/eval_test/eval_result.json`. The artifact checks the checkpoint's embedded config
296
+ against the frozen hash (`checkpoint_config_hash_checked: true`, `config_drift: false`).
297
+
298
+ **Measured numbers:**
299
+
300
+ | Metric | Value | Key path |
301
+ |---|---|---|
302
+ | **pooled IoU** | **0.8122** | `metrics.pooled.iou` |
303
+ | **macro IoU** | **0.8457** | `metrics.macro.miou` |
304
+ | **pooled F1** | **0.8964** | `metrics.pooled.f1` |
305
+ | pooled `miou` / precision / recall | 0.9007 / 0.9195 / 0.8745 | `metrics.pooled.*` |
306
+ | macro F1 / `iou` / precision / recall | 0.7962 / 0.7180 / 0.8506 / 0.7757 | `metrics.macro.*` |
307
+ | confusion | tp 5,978,997 Β· fp 523,658 Β· fn 858,407 Β· tn 126,856,666 | `metrics.pooled.*` |
308
+ | n pixels | 134,217,728 | `metrics.pooled.n_pixels` |
309
+ | images with change | 935 / 2,048 | `n_images_with_change` |
310
+ | mean change fraction | 0.0509 (p50 0.0, p90 0.197205, max 0.684937) | `metrics.mean_change_fraction`, `change_fraction_quantiles` |
311
+ | wall time | 55.359 s | `metrics.seconds` |
312
+
313
+ **Acceptance status:** **VERIFIED** and accepted (shipped). This is the **only** headline metric in the
314
+ project carrying the `VERIFIED` tag β€” measured against a single, immutable public test split with a
315
+ frozen threshold (`docs/BENCHMARKS.md` Β§1.2).
316
+
317
+ **Limitations.** Pooled and macro figures diverge (IoU 0.8122 vs 0.8457; F1 0.8964 vs 0.7962), and the
318
+ corpus is heavily zero-change (`p50` change fraction 0.0; only 935 of 2,048 images contain change). No
319
+ cross-dataset evaluation was run.
320
+
321
+ ---
322
+
323
+ ### 7.2 `change_vqa` β€” change question answering head
324
+
325
+ **Kind:** trained head Β· **File:** `change_vqa/head.pt` Β· **Bytes:** 5,822,809 Β· **sha256:**
326
+ `cfae5e43b97ca930f568dc5b8ae4f36b24e9ff717af226159802206ffd63a82a` Β· **Parameters:** 1,453,912
327
+
328
+ **Architecture.** `change_vqa_head_v1` β€” a **two-stage** reasoning head, not a generative decoder
329
+ (`training/change_vqa/model.py`). Stage 1 maps the change representation to a **class-wise change
330
+ estimate** β€” 6 magnitudes + 6 signed deltas + 1 global fraction = **13 outputs**
331
+ (`N_ESTIMATOR_OUTPUTS = 2 Γ— N_CHANGE_CLASSES + 1`), supervised by `label1`/`label2`. Stage 2
332
+ concatenates that estimate with a question encoding and predicts one of **19 answers** (`N_ANSWERS`).
333
+ The estimator's outputs are *also* emitted as evidence, so an answer arrives with its own audit trail.
334
+ Modules: an `estimator` MLP (`change_feature_dim β†’ 256 β†’ 13`), a `change_trunk`, a `question_trunk`
335
+ (text feature + question-type embedding + temporal embedding) and an `answer_head`
336
+ (`fused β†’ 512 β†’ 256 β†’ 19`).
337
+
338
+ **Hyperparameters** (module constants; `base.yaml` β†’ `training:` where applicable):
339
+
340
+ | Parameter | Value | | Parameter | Value |
341
+ |---|---|---|---|---|
342
+ | `ARCHITECTURE_VERSION` | `change_vqa_head_v1` | | `trunk_dim` / `text_dim` | 512 / 256 |
343
+ | `dropout` | 0.10 | | `qtype_embed_dim` / `temporal_embed_dim` | 32 / 8 |
344
+ | estimator outputs | 13 (2 Γ— 6 + 1) | | answer space | 19 |
345
+ | `seed` | 42 | | `epoch_selected` | 8 (on Val answer accuracy) |
346
+ | `stop_reason` | `early_stopping` | | | |
347
+
348
+ **Training data.** CDVQA (`dataset_id: cdvqa`), feature specs `change_feat_v1`,
349
+ `change_cache_spec: c801326f85a185f8`, `text_cache_spec: d2801ea1a314354a`,
350
+ `preprocessing_version: change_vqa_preproc_v1`. The head's change features are backed by the **frozen**
351
+ STANet change detector β€” `frozen_dependency.path: artifacts/change/levir_change_v001/head.pt`, sha256
352
+ `c5ef31277b67aa01…`, `verified_byte_exact_vs_local: true` (the *same* artifact as Β§7.1). Trained on an
353
+ external GPU (Kaggle) β€” see Β§13.
354
+
355
+ **Evaluation protocol.** Two held-out test sets, `test` (n = 39,686) and `test2` (n = 31,036).
356
+ `epoch_selected` was chosen on **Val answer accuracy = 0.700018**; `PROMOTION.json` records 93 checks
357
+ passed, 0 failed, 0 unverified. Source: `artifacts/change_vqa/run/PROMOTION.json` β†’ `verification`.
358
+
359
+ **Measured numbers:**
360
+
361
+ | Metric | `test` | `test2` |
362
+ |---|---|---|
363
+ | **accuracy** | **0.697626367** | **0.651469262** |
364
+ | **macro F1** | **0.378373275** | **0.372308516** |
365
+ | global-majority baseline | 0.311546 | 0.178728 |
366
+ | n scored | 39,686 | 31,036 |
367
+
368
+ `mask_gain: 0.0`. `metric_ruling: "OPEN β€” the plan leaves the accuracy/macro-F1 interpretation
369
+ owner-gated. No official aggregate metric is asserted here."`
370
+
371
+ **Acceptance status:** MEASURED on two test sets; **ruling OPEN**. `PROMOTION.json` is explicit that
372
+ promotion "records provenance and wires the serving path. It does not itself confer VERIFIED status."
373
+
374
+ **Limitations.** The wide accuracy–macro-F1 gap (0.697626 vs 0.378373) is the signature of class
375
+ imbalance: accuracy is dominated by frequent answers while macro-F1 exposes weak rare-class performance
376
+ (`docs/LIMITATIONS.md` Β§1.4). Confidence at this head is **raw, not calibrated** (`method` reads
377
+ `"uncalibrated"`). The two test sets disagree (0.697626 vs 0.651469), so quoting one alone is selective.
378
+
379
+ ---
380
+
381
+ ### 7.3 `optical_sar` β€” CROMA-base fusion head
382
+
383
+ **Kind:** trained head (production) Β· **File:** `optical_sar/head.pt` Β· **Bytes:** 14,427,457 Β·
384
+ **sha256:** `785815729a3a39fc34dc41894efaf00d8739365d970a3f830a326e68ae888dab`
385
+
386
+ **Architecture.** A fusion head over **frozen CROMA-base** features. CROMA emits three 768-d GAP vectors
387
+ per sample (`optical_GAP`, `SAR_GAP`, `joint_GAP`); the head concatenates them with the availability
388
+ masks β€” `optical_mask (B,12)` and `sar_mask (B,2)` β€” into a **(B, 2318)** tensor (`3Γ—768 + 12 + 2`), then
389
+ `LayerNorm β†’ Linear(2318 β†’ 512) β†’ GELU β†’ Dropout(0.2) β†’ Linear(512 β†’ 19)`. Source:
390
+ `specialists/optical_sar/fusion_head.py`. The **availability mask is consumed by the head, not by
391
+ CROMA** (finding C-1): handing CROMA the mask would invite it to reconstruct missing channels β€” the
392
+ fabrication the sensor adapter exists to prevent.
393
+
394
+ **Hyperparameters** (`base.yaml` β†’ `croma:` and `fusion:`):
395
+
396
+ | Parameter | Value | | Parameter | Value |
397
+ |---|---|---|---|---|
398
+ | `croma.checkpoint_file` | `CROMA_base.pt` (rev `0dd28e3d633b`) | | `croma.image_resolution` | 120 (`% 8 == 0`) |
399
+ | `croma.encoder_dim` | 768 | | `croma.optical_channels` / `sar_channels` | 12 / 2 |
400
+ | `croma.modalities_used` | `[optical, sar, joint]` | | `fusion.input_dim` | 2318 |
401
+ | `fusion.hidden_dim` | 512 | | `fusion.dropout` | 0.2 |
402
+ | `fusion.num_classes` | 19 (BigEarthNet CLC) | | channel/band dropout | **mandatory** (freeze Β§2.5) |
403
+
404
+ **Training data.** reBEN / BigEarthNet-S1 (`data/bigearthnet_v2/`, 480,038 rows in `metadata.parquet`;
405
+ `docs/OWNER_DECISIONS_2026-09-23.md` D-11). The extraction used the **`require_single_label` policy**
406
+ (`n_skipped_by_policy: 0`), which preserves the frozen single-label 19-class softmax but **changes the
407
+ evaluation population** (see limitations). The A/B arm decision was made separately on
408
+ `best_val_accuracy` β€” **A 0.837100 vs B 0.839100**, floor 0.0285 β†’ **Arm A retained** (owner ruling
409
+ R-14; `docs/PHASE12_115_METRIC_COMPUTED.md` Β§5).
410
+
411
+ **Evaluation protocol.** The pre-registered 11.5 metric: fusion-head accuracy and macro-F1 over the
412
+ 19-class label space on the held-out `test` split, n = 4,000, cache arm A
413
+ (`docs/PHASE14_CROMA_NORMALISATION_CHANGE.md` Β§4). Computed by a **separate, later, read-only** tool
414
+ (`scripts/eval_fusion_115.py`); the trainer deliberately never opens the test split
415
+ (`pre_registered_metric_computed = false` in every run record). Source:
416
+ `artifacts/optical_sar/fusion_head_production_v001/pre_registered_115_metric.json`.
417
+
418
+ **Measured numbers:**
419
+
420
+ | Metric | Value | Key path |
421
+ |---|---|---|
422
+ | **accuracy** | **0.931** | `accuracy` |
423
+ | **macro F1** | **0.434161** | `macro_f1` |
424
+ | loss | 0.254592 | `loss` |
425
+ | n scored / classes | 4,000 / 19 | `n_scored` / `num_classes` |
426
+ | classes present | `[0,2,3,4,5,6,7,8,9,10,12,13,17,18]` | `classes_present` |
427
+ | classes absent | `[1,11,14,15,16]` | `classes_absent` |
428
+ | macro-F1 denominator | all 19 slots (absent classes contribute 0.0) | `macro_f1_denominator` |
429
+ | present-only macro-F1 (diagnostic) | 0.589218 | `docs/PHASE12_115_METRIC_COMPUTED.md` Β§3.5 |
430
+
431
+ The majority class holds 2,264 / 4,000 = **0.566**, so 0.931 is not a constant predictor. Per-class F1
432
+ (`_per_class_f1`) shows a wide spread: one class is perfect (1.000), while classes **5** and **6** are
433
+ **present but score 0.000** β€” genuine per-class failures, not absent-class artifacts. The median of the
434
+ 14 present classes is **0.6857** against an accuracy of 0.931 β€” the signature of prediction dominated by
435
+ frequent classes (`docs/PHASE12_115_METRIC_COMPUTED.md` Β§3.4–§3.5).
436
+
437
+ **Acceptance status:** MEASURED; **ruling OPEN**. Whether 0.931/0.434 constitutes a Phase 12 *pass* is
438
+ the **metric-of-record ruling**, which "has not been made, and it is not engineering's to make." The
439
+ artifact's own `is_deciding_statistic: false` and advisory text state that it "selects no head, ranks
440
+ nothing and compares no arms."
441
+
442
+ **Limitations.** (1) **Never quote accuracy alone** β€” 0.931 travels with macro-F1 0.434161. (2) The
443
+ metric describes a **single-label subset**, not multi-label reBEN: single-label patches are 17.57 % of
444
+ the corpus (96,537 / 549,488), and under this policy the rarest class survives as **1 patch** (a
445
+ 59,204 : 1 imbalance). It **may not** be presented as a multi-label BigEarthNet/reBEN result, nor as
446
+ comparable to published BigEarthNet numbers, nor as a statement about all 19 classes β€” 5 have no test
447
+ samples here. (3) The live service returns a **bare class index** (`class_18`), not a CLC label
448
+ (`docs/LIMITATIONS.md` Β§1.6).
449
 
450
+ ---
451
+
452
+ ### 7.4 `grounding` β€” RemoteCLIP grounding head
453
+
454
+ **Kind:** trained head Β· **File:** `grounding/head.pt` Β· **Bytes:** 12,639,041 Β· **sha256:**
455
+ `93432f7034be91a8ffd9c1a84e3eeec00bed7832c043fe7f83d2be230284c6bb` Β· **Parameters:** 1,052,677
456
+ (`docs/OWNER_DECISIONS_2026-09-23.md` D-4, measured against the real checkpoint).
457
+
458
+ **Architecture.** A text-conditioned per-cell box regressor over **frozen RemoteCLIP ViT-B/32** tokens.
459
+ At 224 px the patch grid is **7Γ—7 = 49** tokens of projected dim **512**; the text embedding (512) is
460
+ broadcast to every cell, and each cell's feature is `concat([patch, text, patch*text, global_pool]) =
461
+ 4 Γ— 512 = 2048`. The head is `Linear(2048 β†’ 512) β†’ LayerNorm β†’ Dropout(0.10) β†’ Linear(512 β†’ 5)`,
462
+ emitting `[tx, ty, tw, th, obj]` per cell. Boxes are **cell-relative** (YOLO-style), and exactly **one
463
+ cell per target** is positive β€” the one containing the ground-truth box centre. Source:
464
+ `specialists/grounding/head.py`.
465
+
466
+ **Hyperparameters** (`base.yaml` β†’ `grounding:`, `grounding_head:`, `grounding_training:`):
467
+
468
+ | Parameter | Value | | Parameter | Value |
469
+ |---|---|---|---|---|
470
+ | `image_size` | 224 (`resolution_frozen: true`) | | `model_name` | `ViT-B-32` |
471
+ | `encoder_projected_dim` | 512 (width 768 β†’ projected 512, P7-1) | | `nms_iou` | 0.50 |
472
+ | `max_candidates` | 20 | | `confidence_threshold` | 0.40 |
473
+ | `benchmark_box_scale` | 100.0 (VRSBench 0–100 β†’ stored 0–1) | | `head.feature_dim` | 2048 |
474
+ | `head.hidden_dim` | 512 | | `head.dropout` | 0.10 |
475
+ | `head.positive_confidence_weight` | 20.0 (1 positive in 49) | | `head.decode` | `cell_relative` |
476
+ | training lr / batch / epochs | 1e-4 / 16 / 20 | | training wd / warmup / grad_clip | 1e-4 / 0.05 / 1.0 |
477
+ | training `val_fraction` | 0.10 | | loss weights box/giou/conf | 0.5 / 0.3 / 0.2 |
478
+
479
+ **Training data.** VRSBench (`training/data/vrsbench/`), 16,159 eval records, all images present
480
+ (`docs/OWNER_DECISIONS_2026-09-23.md` D-11). Resolution frozen at **224** by a pre-registered decision
481
+ (see protocol).
482
+
483
+ **Evaluation protocol.** Full VRSBench eval split, **16,159 / 16,159 records**, resolution 224, CPU
484
+ (canonical and matched6 artifacts record `device: cpu`, torch 2.14.0+cpu). Grounding is reported under
485
+ **two protocols** β€” *canonical* (config default `top_k = 20`) and *matched6* (`top_k = 6`, matching the
486
+ zero-shot baseline's mean 5.99 candidates) β€” and **two decode variants** β€” `head_threshold` (score
487
+ threshold 0.40) and `head_argmax`. `head_decode`: `nms_iou 0.5`, `score_threshold 0.4`.
488
+
489
+ **Measured numbers** (canonical: `…/eval_result_canonical.json`; matched6: `…/eval_result_matched6.json`):
490
+
491
+ | Protocol / decode | mean best IoU | recall@0.10 | recall@0.25 | recall@0.50 |
492
+ |---|---|---|---|---|
493
+ | **canonical** `head_threshold` | **0.2838** | 0.6882 | 0.5047 | **0.2198** |
494
+ | **canonical** `head_argmax` | **0.1215** | 0.3183 | 0.2088 | 0.0795 |
495
+ | **canonical** `zero_shot_matched` | **0.0972** | 0.3298 | 0.1188 | 0.0234 |
496
+ | **matched6** `head_threshold` | **0.2566** | 0.6315 | 0.4545 | **0.1938** |
497
+ | **matched6** `head_argmax` | **0.1215** | 0.3183 | 0.2088 | 0.0795 |
498
+ | **matched6** `zero_shot_matched` | **0.0972** | 0.3298 | 0.1188 | 0.0234 |
499
+
500
+ Latency: `head_threshold` 2.205 ms/image (canonical) / 2.158 (matched6); `head_argmax` 0.655 / 0.652;
501
+ `zero_shot_matched` 17.9 s / 15.5 s total. The zero-shot decode is
502
+ `threshold_box_plus_local_maxima`, `delta 0.02`, `top_k 5`, mean 5.99 candidates/image. The artifact's
503
+ `phase7_reference` records the zero-shot floor `mean_best_iou 0.0972`, `recall_at_0.50 0.0234`
504
+ (`docs/PHASE7_RESOLUTION_DECISION.md`).
505
+
506
+ **The 224-vs-448 decision (pre-registered, then confirmed).** The rule was fixed *before* the result was
507
+ seen: *448 wins if Recall@0.5 improves by β‰₯ 0.05 absolute OR mean best IoU improves by β‰₯ 0.05 absolute;
508
+ 224 wins otherwise.* Result: **224 WINS** β€” mean best IoU gain βˆ’0.0147, recall@0.5 gain βˆ’0.0022, at
509
+ 1.59Γ— the latency. Paired over the identical 16,159 samples: mean paired diff βˆ’0.0147, 95 % CI
510
+ [βˆ’0.0160, βˆ’0.0134], **t = βˆ’22.63**; 448 better on 8.5 %, worse on **20.9 %**. The artifact records
511
+ `rule_changed_since_preregistration: false` (`docs/PHASE7_RESOLUTION_DECISION.md`).
512
+
513
+ **Acceptance status:** MEASURED under two protocols; shipped. The trained head is the **production
514
+ default** (owner decision D-4); zero-shot is an **explicit, labelled fallback**, and the system must
515
+ never silently claim `trained` while running zero-shot.
516
+
517
+ **Limitations.** Absolute IoU is low (0.2838 canonical / 0.2566 matched6) β€” the head clearly beats the
518
+ zero-shot baseline (0.0972) but 0.28 is not "solved". The number is **protocol-sensitive**: an absolute
519
+ value is meaningless without its protocol and decode variant. `head_argmax` (0.1215) is **not**
520
+ apples-to-apples with the multi-box baseline (mean best IoU is a max over predictions, so 1 box vs ~6
521
+ flatters the head). Boxes are image-relative, not geodetic. 448 was rejected at the *zero-shot* level;
522
+ whether a *learned* head has the same resolution sensitivity is `UNKNOWN β€” not established from the
523
+ available evidence` (`docs/PHASE7_RESOLUTION_DECISION.md`).
524
+
525
+ ---
526
+
527
+ ### 7.5 `router` β€” intent adapter over frozen MiniLM
528
+
529
+ **Kind:** trained adapter Β· **File:** `router/adapter.pt` Β· **Bytes:** 211,961 Β· **sha256:**
530
+ `8527c3ed28a293e13293d48601d48e3ceafa137b9acabddaf5de31a58a509b5c` Β· **Parameters:** ~50,822
531
+
532
+ **Architecture.** The only trainable part of the router (`router/adapter.py`, `IntentAdapter`):
533
+
534
+ ```
535
+ embedding (384) β†’ LayerNorm β†’ Linear(384 β†’ 128) β†’ GELU β†’ Dropout(0.10)
536
+ β”œβ”€β”€ task_head Linear(128 β†’ 6) # vqa/caption/grounding/change/optical_sar/unsupported
537
+ β”œβ”€β”€ modality_head Linear(128 β†’ 4)
538
+ β”œβ”€β”€ temporal_head Linear(128 β†’ 1) # logit; P(yes) = sigmoid(logit)
539
+ β”œβ”€β”€ spatial_head Linear(128 β†’ 1)
540
+ └── language_head Linear(128 β†’ 1)
541
+ ```
542
+
543
+ Heads are initialised with small-std weights (`std 0.02`, zero bias) so the initial sigmoid sits near
544
+ 0.5 and the binary heads do not start saturated. The adapter does **not** back-propagate into MiniLM.
545
+
546
+ **Hyperparameters** (`base.yaml` β†’ `router:` and `router.training:`):
547
+
548
+ | Parameter | Value | | Parameter | Value |
549
+ |---|---|---|---|---|
550
+ | `model` | `all-MiniLM-L6-v2` (rev `1110a243fdf4`) | | `max_length` | 128 (ceiling is 256) |
551
+ | `embedding_dim` | 384 | | `hidden_dim` | 128 |
552
+ | `dropout` | 0.10 | | `num_tasks` | 6 |
553
+ | `confidence_threshold` | 0.70 | | `epochs` / `batch_size` | 60 / 64 |
554
+ | `learning_rate` / `weight_decay` | 0.001 / 0.01 | | loss weights task/modality/binary | 1.0 / 0.3 / 0.5 |
555
+ | `val_ratio` | 0.15 | | `hard_negatives_to_test` | `true` |
556
+
557
+ **Training data.** A synthetic query corpus: **576** queries in **54** groups
558
+ (`artifacts/router/threshold_sweep_val.json` β†’ `corpus_total`, `corpus_groups`), split
559
+ `train 410 / val 86 / test 80`. Splits are **by group** (template / hard-negative family), never by
560
+ example, so template variants cannot leak across the boundary (F4-3). Hard-negative families are placed
561
+ in the **test** split so their accuracy measures generalisation, not memorisation. The encoder is frozen,
562
+ so embeddings are cached and the adapter trains on cached vectors β€” measured on CPU: **20 epochs over
563
+ 4,096 Γ— 384 in 0.28 s** (F4-2).
564
+
565
+ **Evaluation protocol.** A **validation-only threshold sweep** over 50 thresholds `0.50 … 0.99`, on val
566
+ **n = 86**, `select_by: covered_accuracy`. Source: `artifacts/router/threshold_sweep_val.json`. The test
567
+ split was **NOT touched** (`test_split_touched: false`, `n_test_examples_scored: 0`).
568
+
569
+ **Measured numbers:**
570
 
571
+ | Metric | Value | Key path |
572
+ |---|---|---|
573
+ | **overall ungated accuracy** | **0.965116** | `overall_ungated_accuracy` |
574
+ | n val | 86 | `n_val` |
575
+ | corpus total / groups | 576 / 54 | `corpus_total` / `corpus_groups` |
576
+ | split sizes | train 410 / val 86 / test 80 | `split_sizes` |
577
+ | shipped threshold | 0.70 | `shipped_threshold` |
578
+ | shipped row (thr 0.70) | coverage 0.848837 Β· covered acc 0.972603 Β· fallback 0.151163 Β· n_covered 73 | `shipped_row` |
579
+ | selected row (thr 0.76) | coverage 0.790698 Β· covered acc 1.0 Β· fallback 0.209302 Β· n_covered 68 | `selected` |
580
+ | val per-task support | caption 8 Β· change 20 Β· grounding 14 Β· optical_sar 10 Β· unsupported 19 Β· vqa 15 | `val_task_counts` |
581
+ | hard negatives in val | 0 | `hard_negatives_in_val` |
582
+ | adapter config hash | `615478910dc266bf` | `adapter_config_hash` |
583
+ | encoder | 22,713,216 params, `max_length` 128, rev `1110a243fdf4` | `adapter_encoder` |
584
+
585
+ **Acceptance status:** MEASURED (val only); shipped; **test split NOT RUN**.
586
+
587
+ **Limitations.** The artifact is explicit that this is **not** a calibration and **not** a test result:
588
+ *"corpus-limited: val n=86 vs plan >=500. This is NOT a calibration β€” the corpus is synthetic and too
589
+ small (min per-class support 8, caption) and val carries 0 hard negatives (hn_* families are held out to
590
+ TEST by design). Selecting a threshold here yields a justified default, not a calibrated value."*
591
+ `plan_min_val_queries: 500` and `plan_min_hard_negatives: 100` are both unmet. The number is
592
+ **ungated** accuracy, and the router has known residuals β€” e.g. *"What is the new runway?"* reads
593
+ `change`, not `vqa` (`docs/LIMITATIONS.md` Β§2).
594
+
595
+ ---
596
+
597
+ ### 7.6 `vlm` β€” SmolVLM LoRA adapter (USABLE_VERIFIED, ACCEPTANCE-REJECTED)
598
+
599
+ **Kind:** LoRA adapter (PEFT) Β· **File:** `vlm/adapter_model.safetensors` Β· **Bytes:** 34,798,048 Β·
600
+ **sha256:** `07c76a75fa04624880ed7730590f5fdd7b145a8232e3c0af411c3c545a5adf5e` Β· **Trainable params:**
601
+ 8,683,520 (1.6823 % of the 516,165,824-param base)
602
+
603
+ **Architecture.** A **PEFT LoRA** adapter (`r = 16`, `alpha = 32`, `dropout = 0.05`) on the
604
+ **text-model projections** of frozen `HuggingFaceTB/SmolVLM-500M-Instruct` (rev `a7da5b986cb5`). Target
605
+ modules: `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` across **224** modules.
606
+ `trainable_subtrees` is exactly `{"model.text_model": 8683520}` β€” the **vision tower was untouched**
607
+ (86,433,024 frozen) and the connector (11,796,480) is frozen too. Precision: **fp16**.
608
+
609
+ **Hyperparameters** (`base.yaml` β†’ `training:`, `vlm:`; `phase6_closure.json`):
610
+
611
+ | Parameter | Value | | Parameter | Value |
612
+ |---|---|---|---|---|
613
+ | `checkpoint` | `SmolVLM-500M-Instruct` (rev `a7da5b986cb5`) | | `lora_rank`/`alpha`/`dropout` | 16 / 32 / 0.05 |
614
+ | LoRA target modules | 224 | | `precision` | fp16 (T4 is SM 7.5 β†’ not bf16; C-6) |
615
+ | `vlm_batch_size` / `grad_accum` | 2 / 8 | | `vlm_learning_rate` / `vlm_epochs` | 2e-4 / 1 |
616
+ | `weight_decay` / `warmup_ratio` | 0.01 / 0.05 | | `gradient_checkpointing` | `true` |
617
+ | `processor_longest_edge` | 512 (default 2048 splits a tile into 17 sub-images β€” F5-2) | | `do_sample` / `temperature` | `false` / 0.0 |
618
+ | `max_new_tokens` | 128 | | `max_images_per_call` / `seed` | 1 / 42 |
619
 
620
+ **Training data.** BigEarthNet-derived presence questions (`kind: bigearthnet_smolvlm_lora`). Trained on
621
+ an external GPU (Kaggle, T4) β€” see Β§13. The adapter directory holds **14 files** verified against its
622
+ own `ARTIFACT_SHA256SUMS.json`.
623
 
624
+ **Evaluation protocol.** A frozen **1,000-question test subset** (`available_per_split {val: 6750,
625
+ test: 7772}`, subset n = 1000, 19 classes summing to 1000). The pre-registered acceptance rule is
626
+ **v002**: V1 requires aggregate test delta β‰₯ +5.00 pp; V2 (a per-class guardrail) fails a class with
627
+ `n β‰₯ 20` questions iff it **both** lost β‰₯ 4 questions **and** has `z β‰₯ 1.96`. Source:
628
+ `artifacts/vlm/phase6_closure.json` β†’ `why_acceptance_rejected`, `why_usable_verified`.
629
 
630
+ **Measured numbers:**
631
+
632
+ | Metric | Value | Key path |
633
  |---|---|---|
634
+ | **exact_match** | **0.963** | `why_usable_verified.adapted_test.exact_match` |
635
+ | **F1** | **0.96432** | `why_usable_verified.adapted_test.f1` |
636
+ | precision / recall | 0.963391 / 0.965251 | `…adapted_test.precision` / `.recall` |
637
+ | confusion | tp 500 Β· fp 19 Β· tn 463 Β· fn 18 | `…adapted_test.confusion` |
638
+ | n | 1,000 | `…adapted_test.n` |
639
+ | aggregate test delta | **+49.50 pp** (46.80 β†’ 96.30) | `why_usable_verified.aggregate_test_delta_pp` |
640
+
641
+ **Why it is *usable and verified*.** Gate D reproduced Run 1's adapted-test control **exactly**
642
+ (`exact_match 0.963`, `f1 0.9643201542912246`, identical confusion), proving the local artifact *is*
643
+ Run 1's adapter and that CPU/fp32 reproduces the Kaggle T4 endpoint. Gate Aβ€³ proved subset identity
644
+ without a model. The 14-file manifest check is **clean** (0 missing, 0 mismatched, 0 extra) and the
645
+ adapter **loads through the production path** (`PeftModel.from_pretrained`).
646
+
647
+ **Why it is *acceptance-rejected*.** V1 **passes** (+49.50 pp β‰₯ +5.00), but V2 **fails**: class
648
+ **Mixed forest** (`n = 33`) goes `100.00 β†’ 87.8788 pp`, a drop of **12.1212 pp**, `lost_questions 4`,
649
+ `z 2.1335` β€” failing **both** halves of v002. Per item V, a complete run that fails V2 is `REJECTED`.
650
+ The rejection is narrow (1 of 19 classes fails; 11 improved, 5 held) and is **not a split artefact** β€”
651
+ the same class also degraded on val in Run 1 (drop 6.4516 pp, n = 31). Residual risk, reported not
652
+ resolved: the verdict rests on 4 questions in one class of 33, the unfloored minimum-size exposure
653
+ recorded at `PHASE6_AUDIT_AND_CONTRACT.md` Β§8.6.
654
+
655
+ **Acceptance status:** **USABLE_VERIFIED** and **ACCEPTANCE-REJECTED** β€” both true, answering different
656
+ questions. **`USABLE_VERIFIED` β‰  `ACCEPTANCE-ACCEPTED`.** The deployed caption/VQA path uses the
657
+ **unadapted** model; the adapter is enabled only via the `SATQUERY_VLM_ADAPTER` environment variable
658
+ (`specialists/vqa/model.py` β†’ `ADAPTER_ENV_VAR`).
659
+
660
+ **Limitations and traps.** (1) The adapter is **not** accepted for production use. (2) The
661
+ `Mixed forest` regression is **not resolved**. (3) `adapter_sha256` names **two different values** and
662
+ they are not interchangeable β€” a **tree hash** over the weight map (`5c6b8631…`, from
663
+ `training/vlm/artifact.py`) versus the **file** sha256 of `adapter_model.safetensors` (`07c76a75…`, from
664
+ `specialists/vqa/model.py::_adapter_sha256`); comparing one against the other produces a false
665
+ "artifact was altered" conclusion. (4) The promoted adapter is **not** `checkpoint-2000` β€” the three
666
+ weight files have three distinct digests (top-level `07c76a75…`, `checkpoint-1500` `7273588e…`,
667
+ `checkpoint-2000` `bf249943…`). (5) The adapter's canonical path is under `.scratch/`; it is
668
+ **reconstructible** from `phase6_realbundle.zip` and verified against the two digests above.
669
+
670
+ ---
671
+
672
+ ## 8. Full measured-performance table
673
+
674
+ Every row names its source artifact and the exact key path. The `n` and `split` columns are part of the
675
+ claim, not decoration: a metric without its population is not a result. All 20 numeric claims are
676
+ checked against these files by [`tools/verify_readme_metrics.py`](tools/verify_readme_metrics.py); its
677
+ output (`ALL CLAIMS VERIFIED`) is committed as
678
+ [`tools/readme_metrics_report.txt`](tools/readme_metrics_report.txt).
679
+
680
+ | Capability | Metric | Value | Split / protocol | n | Source β†’ key path | Status |
681
+ |---|---|---|---|---|---|---|
682
+ | Change | pooled IoU | **0.8122** | LEVIR-CD-256 test, thr 0.50 | 2,048 | `change/eval_test/eval_result.json` β†’ `metrics.pooled.iou` | **VERIFIED** |
683
+ | Change | macro IoU | **0.8457** | same | 2,048 | `…` β†’ `metrics.macro.miou` | **VERIFIED** |
684
+ | Change | pooled F1 | **0.8964** | same | 2,048 | `…` β†’ `metrics.pooled.f1` | **VERIFIED** |
685
+ | Grounding | mean best IoU | **0.2838** | VRSBench canonical (thr, top_k 20) | 16,159 | `grounding/…/eval_result_canonical.json` β†’ `results.head_threshold.mean_best_iou` | MEASURED (2 protocols) |
686
+ | Grounding | recall@0.5 | **0.2198** | canonical | 16,159 | `…canonical.json` β†’ `results.head_threshold.recall.0.50` | MEASURED (2 protocols) |
687
+ | Grounding | mean best IoU | **0.2566** | VRSBench matched6 (thr, top_k 6) | 16,159 | `…matched6.json` β†’ `results.head_threshold.mean_best_iou` | MEASURED (2 protocols) |
688
+ | Grounding | recall@0.5 | **0.1938** | matched6 | 16,159 | `…matched6.json` β†’ `results.head_threshold.recall.0.50` | MEASURED (2 protocols) |
689
+ | Grounding | head-argmax IoU | **0.1215** | canonical (argmax) | 16,159 | `…canonical.json` β†’ `results.head_argmax.mean_best_iou` | MEASURED |
690
+ | Grounding | zero-shot baseline IoU | **0.0972** | canonical (no head) | 16,159 | `…canonical.json` β†’ `results.zero_shot_matched.mean_best_iou` | MEASURED (baseline) |
691
+ | Optical-SAR | accuracy | **0.931** | held-out test, 19 classes | 4,000 | `optical_sar/…/pre_registered_115_metric.json` β†’ `accuracy` | MEASURED, ruling **OPEN** |
692
+ | Optical-SAR | macro F1 | **0.434161** | same | 4,000 | `…` β†’ `macro_f1` | MEASURED, ruling **OPEN** |
693
+ | Change-VQA | accuracy | **0.697626** | `test` | 39,686 | `change_vqa/run/PROMOTION.json` β†’ `verification.test_accuracy` | MEASURED, ruling **OPEN** |
694
+ | Change-VQA | macro F1 | **0.378373** | `test` | 39,686 | `…PROMOTION.json` β†’ `verification.test_macro_f1` | MEASURED, ruling **OPEN** |
695
+ | Change-VQA | accuracy (2nd set) | **0.651469** | `test2` | 31,036 | `…PROMOTION.json` β†’ `verification.test2_accuracy` | MEASURED, ruling **OPEN** |
696
+ | Change-VQA | macro F1 (2nd set) | **0.372309** | `test2` | 31,036 | `…PROMOTION.json` β†’ `verification.test2_macro_f1` | MEASURED, ruling **OPEN** |
697
+ | VLM (adapted) | exact_match | **0.963** | frozen 1,000-question subset | 1,000 | `vlm/phase6_closure.json` β†’ `why_usable_verified.adapted_test.exact_match` | MEASURED, **ACCEPTANCE-REJECTED** |
698
+ | VLM (adapted) | F1 | **0.96432** | same | 1,000 | `…phase6_closure.json` β†’ `…adapted_test.f1` | MEASURED, **ACCEPTANCE-REJECTED** |
699
+ | Router | overall **ungated** accuracy | **0.965116** | val, corpus-limited | 86 | `router/threshold_sweep_val.json` β†’ `overall_ungated_accuracy` | MEASURED β€” **TEST NOT RUN** |
700
+ | Calibration | ECE before / after | **0.013755 β†’ 0.014929** | val, T = 0.9773 | 16,441 | `calibration_v001.json` β†’ `metrics.ece_before` / `.ece_after` | MEASURED β€” **worse** |
701
+ | System | end-to-end accuracy | β€” | β€” | β€” | β€” | **NOT RUN β€” none exists** |
702
 
703
+ ---
704
+
705
+ ## 9. Calibration β€” a measured negative result
706
 
707
+ Temperature scaling is **enabled** in the frozen configuration (`confidence.temperature_scaling: true`,
708
+ `confidence.calibration_file: calibration_v001.json`) and applied by
709
+ `evidence.confidence.TemperatureCalibration` as `sigmoid(logit(z)/T)` for a scalar `z` and
710
+ `softmax(logits/T)` for a distribution. Source: `artifacts/calibration_v001.json`.
711
 
712
+ | Field | Value | | Field | Value |
713
+ |---|---|---|---|---|
714
+ | `method` | `temperature_scaling` | | `temperature` | **0.9772731820958189** |
715
+ | `fitted_on` / `n_samples` | Val / **16,441** | | `n_classes` / `space` | 19 / `multiclass_logits` |
716
+ | `objective` | `mean_negative_log_likelihood` | | `optimizer` | `golden_section_on_log_temperature` (200 iters, `hit_bound: false`) |
717
+ | NLL before β†’ after | 0.689741 β†’ 0.689631 (Ξ” 0.00011) | | **ECE before β†’ after** | **0.013755 β†’ 0.014929** |
718
+ | `ece_improvement` | **βˆ’0.001174** (negative β‡’ did **not** help) | | `n_bins` | 15 |
719
+ | scope | `change_vqa` only β€” "Other specialists emit their own raw scores and are unaffected." | | `type_mask_applied` | `false` |
720
+ | held-out splits excluded | `[Test, Test2]` | | | |
721
 
722
+ **The honest reading: the ECE got worse.** Temperature scaling reduced the NLL very slightly (0.00011)
723
+ but **increased** the expected calibration error from 0.013755 to 0.014929. It is retained **only
724
+ because it is part of the frozen configuration** β€” not because it helped. This is a measured negative
725
+ result and is reported as one (`docs/BENCHMARKS.md` Β§4.7, `docs/MODELS.md` Β§5).
726
 
727
+ **Two caveats on the number.** The artifact notes that "ECE is bin-count sensitive and is not an
728
+ aggregate score", and the `reliability_diagram` it carries is the **pre-scaling** curve (`ece 0.013755`),
729
+ labelled as such β€” the calibrated curve is **NOT plotted** (`docs/LIMITATIONS.md` Β§3.22).
 
 
 
 
730
 
731
+ ---
732
 
733
+ ## 10. Acceptance status
734
 
735
+ | Artifact | Metrics | Acceptance | Notes |
736
+ |---|---|---|---|
737
+ | `change` | VERIFIED | accepted (shipped) | the only `VERIFIED` headline |
738
+ | `grounding` | measured (2 protocols Γ— 2 decode variants) | shipped | trained head is the production default; zero-shot is a labelled fallback (D-4) |
739
+ | `optical_sar` | measured | **ruling OPEN** | accuracy 0.931 always with macro-F1 0.434161 |
740
+ | `change_vqa` | measured (2 test sets) | **ruling OPEN** | test + test2 both reported |
741
+ | `router` | measured (val only) | shipped; **test NOT RUN** | 0.965116 is validation, ungated, n = 86 |
742
+ | **`vlm`** | usable (exact_match 0.963, F1 0.96432) | **ACCEPTANCE-REJECTED** | deployed path uses the unadapted model |
743
 
744
+ **`USABLE_VERIFIED` β‰  `ACCEPTANCE-ACCEPTED`.** The VLM adapter works and is not promoted. The two
745
+ questions β€” *is this the artifact we trained, and does it work?* versus *did it clear the bar we
746
+ predeclared before looking?* β€” are kept separate on purpose (`docs/PHASE6_CLOSURE.md` Β§1).
747
+
748
+ ---
749
+
750
+ ## 11. Evaluation gaps (stated, not hidden)
751
+
752
+ | Gap | State |
753
+ |---|---|
754
+ | **System-level end-to-end benchmark** | **NOT RUN β€” none exists.** No end-to-end accuracy is claimed. |
755
+ | **Router test split** | **NOT RUN** (`test_split_touched: false`) |
756
+ | **Benchmark adapters** | **NOT RUN** |
757
+ | **End-to-end latency benchmark** | **NOT RUN** (per-specialist latency recorded only incidentally) |
758
+ | **Cross-dataset generalisation** | **NOT RUN** β€” each specialist is evaluated only on its own training-family split |
759
+ | **Human evaluation** | **NOT RUN** |
760
+ | **Robustness / adversarial evaluation** | **NOT RUN** |
761
+ | **Statistical significance for most metrics** | only the grounding 448-vs-224 decision has a paired test with a CI; other per-task numbers are point estimates |
762
+ | **Calibrated reliability curve** | **NOT plotted** |
763
+ | **BigEarthNet label semantics** | the local subset is **100 % single-label** vs the official 1–11 multi-label scheme, so its metrics are **not comparable** to published numbers |
764
+
765
+ ---
766
+
767
+ ## 12. Limitations
768
+
769
+ A condensed catalogue; the full version is [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Per-artifact
770
+ limitations are in Β§7; evaluation gaps in Β§11.
771
+
772
+ **Model quality.** Grounding absolute IoU is low (0.2838 / 0.2566) and **protocol-sensitive**.
773
+ Optical-SAR accuracy is carried by common classes β€” **0.931 with macro-F1 0.434161**. Change-VQA is weak
774
+ on rare classes (0.697626/0.378373 and 0.651469/0.372309). The live optical-SAR service returns a **bare
775
+ class index** (`class_18`), not a CLC label. The VLM adapter is **not accepted**. Calibration made ECE
776
+ **worse**.
777
+
778
+ **Router.** 0.965116 is **validation, ungated, n = 86**, corpus-limited; the test split was **NOT RUN**.
779
+ Known residuals: *"What is the new runway?"* reads `change`; *"How much built-up area was added?"*
780
+ under-triggers `vqa`; with one asset the console *reads* `change` while dispatch correctly falls back to
781
+ `change_vqa` (intentional, but visually surprising).
782
+
783
+ **Evaluation.** No end-to-end benchmark; no cross-dataset, human or robustness evaluation; most metrics
784
+ are point estimates without confidence intervals. The BigEarthNet local subset is **100 % single-label**,
785
+ so its metrics are **not comparable** to published multi-label numbers.
786
+
787
+ **Operational.** Transient tunnel gaps (B-07) β€” a request can hang or return 504; patch prepared but
788
+ **NOT deployed**. `OPEN`. `/api/health` `codespace_name` carries a trailing newline (B-02) β€” cosmetic.
789
+ `OPEN`. Cold start is tens of seconds; single-region, no HA; no database, auth or queue (stateless by
790
+ design).
791
+
792
+ **Packaging and licensing.** **No `LICENSE` file exists** in the source repository. `OPEN`. The six
793
+ artifacts require their pinned backbones, which are **not** redistributed.
794
+
795
+ **Documentation.** `docs/FINAL_DELIVERY_REPORT.md` Β§6 is stale (it lists the bundled EO change pair as
796
+ DEGRADED and B-01 as BLOCKED; both were resolved on 2026-09-25). The original master plan describes a
797
+ **superseded deployment** (Gradio GUI + HF Space + ZeroGPU + Railway); the shipped system is a static
798
+ frontend + Render + Codespace tunnel serving JSON.
799
+
800
+ **Explicit non-claims.** No state-of-the-art claim; no production-readiness claim for model quality; no
801
+ claim that the trained heads generalise beyond their training-family splits; no claim that calibration
802
+ improves confidence; no claim that the VLM adapter is accepted; no end-to-end accuracy claim; no
803
+ robustness claim; no geolocation-accuracy claim; not a safety-, legal- or life-critical tool.
804
+
805
+ ---
806
+
807
+ ## 13. Training summary
808
+
809
+ All six artifacts are small modules on frozen backbones, trained with **seed 42** and recording the
810
+ frozen config hash **`78f1e3700da15aa1`**.
811
+
812
+ | Artifact | Where it trained | Precision | Notable settings |
813
+ |---|---|---|---|
814
+ | `change` | GPU (eval artifact: `device cuda`, torch 2.10.0+cu128) | β€” | STANet-style; ResNet-18 pretrained; PAM; bce 0.5 + dice 0.5 |
815
+ | `change_vqa` | **external GPU (Kaggle)** | fp16 | epoch 8 selected on val answer accuracy 0.700018; early stopping |
816
+ | `optical_sar` | in-project | β€” | arm A retained (A 0.837100 vs B 0.839100, floor 0.0285); 10 runs Γ— 2 arms Γ— 5 seeds |
817
+ | `grounding` | CPU | β€” | 20 epochs; grid 7Γ—7; resolution frozen at 224 by pre-registered test |
818
+ | `router` | **CPU** | β€” | frozen encoder, cached embeddings; 20 epochs / 4,096 vectors in 0.28 s |
819
+ | `vlm` | **external GPU (Kaggle, T4)** | fp16 | LoRA r=16 Ξ±=32; 224 text-projection modules; vision tower untouched |
820
+
821
+ **Precision.** `training.precision: fp16` because the target GPU (T4) is compute capability 7.5 β€” bf16 is
822
+ unavailable there (finding C-6). The loader validates the value is one of `fp16|bf16|fp32`.
823
+
824
+ **Provenance discipline.** `change_vqa` was promoted from a Kaggle export with 93/0/0 verification
825
+ checks and a byte-identical source copy. The VLM adapter's Phase 6 closure was reached **without
826
+ retraining or modifying** the adapter β€” the promoted weights are the end-of-training top-level save, and
827
+ the closure record is generated from the evidence rather than restated. Full procedures:
828
+ [`docs/TRAINING.md`](docs/TRAINING.md); dataset provenance: [`docs/DATASETS.md`](docs/DATASETS.md).
829
+
830
+ ---
831
+
832
+ ## 14. Provenance and verification
833
 
834
  | Item | Location |
835
  |---|---|
836
+ | Byte-exact manifest (generated from disk) | [`models/manifest.json`](models/manifest.json) |
837
+ | Checksums | [`models/checksums.sha256`](models/checksums.sha256) |
838
+ | Metric verification tool / output | [`tools/verify_readme_metrics.py`](tools/verify_readme_metrics.py) Β· [`tools/readme_metrics_report.txt`](tools/readme_metrics_report.txt) |
839
+ | HF release verification (re-downloaded, 6/6 MATCH) | [`HF_RELEASE_VERIFICATION.md`](HF_RELEASE_VERIFICATION.md) |
840
+ | Release manifest (every file, size + sha256) | [`RELEASE_MANIFEST.md`](RELEASE_MANIFEST.md) |
841
+ | Full documentation / repository front page | [`docs/`](docs/) Β· [`README.md`](README.md) |
842
+
843
+ **Verification chain.** (1) `models/manifest.json` and `models/checksums.sha256` are **generated by
844
+ reading the files** (`release/tools/generate_model_manifest.py`); no byte count or hash is typed by hand.
845
+ (2) `tools/verify_readme_metrics.py` walks every quoted metric to its source artifact; result
846
+ **`ALL CLAIMS VERIFIED`** (20/20), with status assertions (VLM `ACCEPTANCE-REJECTED`, router
847
+ `corpus_limited` n = 86, calibration `ece_improvement` negative) confirmed. (3)
848
+ `release/tools/hf_verify.py` re-downloads each artifact over direct HTTPS and hashes the received bytes;
849
+ **6/6 MATCH, 0 failed**. (4) Two artifact digests agree with values recorded independently at
850
+ promotion/fit time (Β§3).
851
+
852
+ **Hugging Face release.** `thundercode/SatQuery` (public), HEAD `bf2779e18fcaa7476b93a48a978f08c108dfdfb7`,
853
+ `lastModified 2026-09-25T21:46:52Z`, 42 files on the Hub. No secret was uploaded; the token used is not
854
+ written into any released file (`HF_RELEASE_VERIFICATION.md` Β§8).
855
+
856
+ **GitHub release.** The curated public repository target is `Anish-lab-blip/SatQuery-AI`; the release
857
+ tree is staged and its links verified. At the time of `release/RELEASE_EXECUTION_CHECKLIST.md` Phase 5,
858
+ the push was **BLOCKED** because the fine-grained token was read-only for repository contents
859
+ (`403 Resource not accessible by personal access token`) β€” an owner action, not a defect in this release.
860
+ Where this card and a live repository disagree, treat the live repository as authoritative for
861
+ publication state and this card as authoritative for artifact identity.
862
+
863
+ ---
864
 
865
+ ## 15. Licence
866
 
867
+ The project ships **no licence file**; a licence must be selected by the owner before public release of
868
+ the *code* (`docs/LIMITATIONS.md` Β§5, `RELEASE_MANIFEST.md`). This is an **OPEN** item. The Hugging Face
869
+ card declares `license: other` because the correct licence has not yet been chosen.
870
 
871
+ **Model weights carry the terms of their backbone licences.** The six artifacts are small modules, but
872
+ they depend on and are intended to be used with: `sentence-transformers/all-MiniLM-L6-v2`;
873
+ `HuggingFaceTB/SmolVLM-500M-Instruct`; `chendelong/RemoteCLIP` (`RemoteCLIP-ViT-B-32.pt`);
874
+ `antofuller/CROMA` (`CROMA_base.pt`); and torchvision ResNet-18 (`IMAGENET1K_V1`). **Backbones are not
875
+ redistributed here.** Consult each backbone's Hugging Face page for the authoritative licence β€” the
876
+ licence labels above are recorded for convenience and should be verified at the source before reuse.
877
+
878
+ ---
879
+
880
+ ## 16. Citation
881
 
882
  If you use this work, cite the project repository:
883
 
 
889
  note = {Public release: https://github.com/Anish-lab-blip/SatQuery-AI}
890
  }
891
  ```
892
+
893
+ **Model card version.** This card documents release **1.0.0** (2026-09-25), frozen config hash
894
+ `78f1e3700da15aa1`. The changelog is [`docs/CHANGELOG.md`](docs/CHANGELOG.md).