thundercode commited on
Commit
6c5f243
·
verified ·
1 Parent(s): edcf7bc

release: add docs/RESEARCH_NOTES.md

Browse files
Files changed (1) hide show
  1. docs/RESEARCH_NOTES.md +629 -115
docs/RESEARCH_NOTES.md CHANGED
@@ -2,147 +2,410 @@
2
 
3
  Engineering findings, negative results, and design decisions that would otherwise be lost. Every one
4
  was learned by **probe or execution**, not assumption, and every one is recorded so it is not
5
- rediscovered.
 
6
 
7
  **Status tags:** `MEASURED` · `RESOLVED` · `REJECTED` · `OPEN` · `ATTEMPTED`.
8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
  ---
10
 
11
- ## 1. Findings that changed the code
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
- ### F4-1 — the MiniLM tokenizer ceiling (`MEASURED`)
 
14
 
15
- The MiniLM tokenizer's own ceiling is **256** (verified by probe). This project truncates to **128** —
16
- a deliberate truncation *well inside* the ceiling, not the model limit. Satellite queries are short;
17
- halving the sequence halves attention cost for no measurable accuracy loss. The encoder **asserts**
18
- `max_length ≤ 256`, because truncating above the ceiling is a **silent no-op**.
19
 
20
- **Consequence:** `router.max_length: 128` with an assertion, rather than a comment.
 
 
21
 
22
- ### F4-2 — the router needs no GPU (`MEASURED`)
 
 
 
 
23
 
24
- The encoder is frozen, so embeddings are **cached** and the 50,822-parameter adapter trains on cached
25
- vectors. **Measured on CPU: 20 epochs / 4,096 vectors in 0.28 s.**
26
 
27
- **Consequence:** the router can be retrained during development without a GPU.
28
 
29
- ### F4-3 — splits must be by group (`MEASURED`)
30
 
31
- Splits are by **group** (template / hard-negative family), never by example. Hard-negative families
32
- are placed in the **test** split so their accuracy measures generalisation rather than memorisation.
33
- Splitting by example would leak template variants across the boundary and inflate the score.
 
 
34
 
35
- ### F5-1 — `AutoModelForVision2Seq` does not exist (`MEASURED`)
36
 
37
- In transformers 5.17.0, `AutoModelForVision2Seq` **does not exist** (it is not merely deprecated);
38
- `AutoModelForImageTextToText` is present.
39
 
40
- **Consequence:** the loader is resolved by **feature detection**, never hardcoded to one class name.
41
 
42
  ### F5-2 — the processor cost overrun is ~17×, not 4× (`MEASURED`)
43
 
44
- The processor's default `longest_edge` is **2048**, which upscales a 512-px tile **4×** and then
45
- splits it (`do_image_splitting=True`) into sub-images:
46
 
47
- | Setting | `pixel_values` | prompt tokens |
48
- |---|---|---|
49
- | default | `(1, 17, 3, 512, 512)` | 1142 |
50
- | pinned (`processor_longest_edge: 512`) | `(1, 1, 3, 512, 512)` | — |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
 
52
- The plan estimated a 4× cost overrun; the real figure is **~17×**. The value **must** be set
53
- explicitly on the processor at construction time, and `core/config.py` now **enforces**
54
- `processor_longest_edge <= image.tile_size` so the pin is a control rather than a comment.
55
 
56
- ### F5-3 — prompts must go through the chat template (`MEASURED`)
 
 
 
57
 
58
- SmolVLM requires one `<image>` token per image in the prompt. Hand-written prompt strings raise
59
- `ValueError`.
60
 
61
- **Consequence:** prompts are always built through `processor.apply_chat_template()`, enforced by the
62
- config key `vlm.prompt_must_use_chat_template: true`.
63
 
64
- ### P7-1 — the RemoteCLIP projected dimension is 512 (`MEASURED`)
65
 
66
- The RemoteCLIP ViT-B/32 transformer width is **768**, but `visual.proj` maps to a **projected** dim of
67
- **512**. The grounding head's per-cell feature is therefore `4 × 512 = 2048`.
 
68
 
69
- **Consequence:** `grounding.encoder_projected_dim: 512` is declared in config so `core/config.py` can
70
- validate the head **without importing torch**, and `specialists/grounding/remoteclip.py` asserts the
71
- same value against the real model at load time.
 
 
 
 
72
 
73
  ### C-1 — the availability mask is consumed by the head, not by CROMA (`MEASURED`)
74
 
75
- CROMA always sees the canonical channel counts (12 optical, 2 SAR). The availability mask is applied
76
- by the **fusion head** (`input_dim = 3 × 768 + 12 + 2 = 2318`), not by the encoder.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
77
 
78
  ### C-6 — T4 is SM 7.5, so training uses fp16, not bf16 (`MEASURED`)
79
 
80
- `training.precision: fp16` because the target GPU (T4) is compute capability 7.5; bf16 is unavailable
81
- there. The loader validates the value is one of `fp16|bf16|fp32`.
 
 
 
 
 
 
82
 
83
  ### C-7 — `image_resolution % 8 == 0` (`MEASURED`)
84
 
85
- CROMA requires `image_resolution % 8 == 0`. The native value **120** yields 225 patches. Enforced at
86
- config load.
 
 
 
 
87
 
88
  ### C-8 — ZeroGPU does not support `torch.compile` (`MEASURED`)
89
 
90
- `torch.compile` must never be enabled on the (historical) ZeroGPU target. Enforced: the loader
91
- **fails startup** if `deployment.torch_compile` is true.
 
 
 
 
 
 
92
 
93
  ### C-9 — STANet hyperparameters are upstream-verified (`MEASURED`)
94
 
95
- Change detection uses a STANet-style architecture with upstream-verified hyperparameters: ResNet-18
96
- encoder, **PAM** self-attention mode, tile 256, threshold 0.50, BCE 0.5 + Dice 0.5.
 
 
 
 
 
 
 
 
 
 
97
 
98
  ---
99
 
100
- ## 2. The grounding resolution decision — a pre-registered rejection (`REJECTED`)
101
 
102
- **Question:** should grounding decode at 448 or 224?
103
 
104
  **Answer: 224. 448 was rejected** — notable because the rejection was *pre-registered* and then
105
- *confirmed* by a paired test over identical samples (n = 16,159):
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
106
 
107
- | Comparison (448 vs 224) | Value |
108
- |---|---|
109
- | mean best IoU | **−0.0147** |
110
- | recall@0.5 | −0.0022 |
111
- | recall@0.10 | −0.0699 |
112
- | recall@0.25 | −0.0243 |
113
- | latency | **1.59×** |
114
- | paired mean difference | −0.0147 |
115
- | paired 95 % CI | **[−0.0160, −0.0134]** |
116
- | paired t | **−22.63** |
117
- | 448 better on | 8.5 % of records |
118
- | 448 worse on | **20.9 %** of records |
119
-
120
- 448 lost on **every** axis. The pre-registered decision rule and the paired test **agree** on 224.
121
- This is the model for how a resolution decision should be made: declared in advance, then tested.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
122
 
123
  ---
124
 
125
- ## 3. The router defect — a real bug, found and fixed (`RESOLVED`)
126
 
127
- ### 3.1 Symptom
128
 
129
- The query *"Where are the built-up areas in this image?"* collapsed to **`vqa`** and answered
130
- **"River"** — instead of routing to `grounding`. A second query, *"Where is the new airport?"*,
131
- behaved the same way.
132
 
133
- ### 3.2 Root cause
134
 
135
  Two functions with different information:
136
 
137
  - **`interpret()`** — produces the console's *reading*; **asset-count-blind** (text only).
138
  - **`chooseTask()`** — performs *dispatch*; **asset-count-aware**.
139
 
140
- The defect was in the dispatch path's handling of spatial/lexical cues, so region queries fell through
141
- to the generic VQA specialist.
142
 
143
- ### 3.3 Fix and verification
 
 
 
 
144
 
145
- The fix was deployed to `SatQuery-Frontend` and validated by **three independent live passes**:
 
 
 
 
 
 
146
 
147
  | Pass | Deployed HEAD | Result |
148
  |---|---|---|
@@ -152,54 +415,121 @@ The fix was deployed to `SatQuery-Frontend` and validated by **three independent
152
 
153
  Both defect queries now dispatch to `grounding`:
154
 
155
- | Query | Run id | Dispatched |
156
- |---|---|---|
157
- | Where are the built-up areas in this image? | `run_467ffa406f22` | `grounding` |
158
- | Where is the new airport? | `run_46980ba55c62` | `grounding` |
 
 
 
 
 
159
 
160
- **24 live runs, 24 correct dispatches, 0 mock nodes.** Screenshots are in
161
- [`../screenshots/`](../screenshots/).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
162
 
163
  ---
164
 
165
- ## 4. The harness false-positive — caught before it could lie (`RESOLVED`)
 
 
166
 
167
  An earlier live-validation harness typed queries with **synthetic CDP key events**, which Chrome
168
- **silently drops when the window lacks OS focus**. The harness therefore dispatched the page's
169
- *default* query and still recorded a "result" — a **false pass**.
 
 
 
 
 
170
 
171
- **Fix:** the current harness **asserts form state before dispatch** (`q_ok`, `obs_ok`, `t0_ok`) and
172
- uses deterministic query entry (`js()` value-set + `type_text()` via CDP `Input.insertText`).
 
173
 
174
- **Independent check:** the earlier 8/8 run was re-examined and confirmed **not** infected — its
175
- answers were query-specific and the query text was embedded in the answers. The failure mode is
176
- recorded because it is exactly the silent false-positive an evaluation harness must never have.
 
 
 
 
 
 
 
 
 
 
 
 
177
 
178
  ---
179
 
180
- ## 5. The `transport_mode: auto` fallthrough (`OPEN`)
181
 
182
  `SATQUERY_TRANSPORT=auto` tries the tunnel, then falls through to the forward path on timeout. The
183
  forward path to a **private** repo returns `302` quickly, but the wake step still consumes
184
  `SATQUERY_WAKE_TIMEOUT_S` (120 s) first — so a worst-case failed request takes ≈ **249 s**
185
- (150 + 120). This is the root shape of the observed transient tunnel gap.
186
 
187
- A patch adding `forward_unavailable` (503) and `upstream_timeout` (504) codes plus the
188
- `codespace_name` `.strip()` fix was authored and verified (`py_compile` clean). **Status: OPEN — the
189
- patch is prepared but NOT deployed.**
 
 
 
 
 
 
 
 
 
 
 
190
 
191
  ---
192
 
193
- ## 6. The `interpret()` / `chooseTask()` asymmetry — intentional (`RESOLVED`)
194
 
195
  For *"What changed between the earlier and later image?"* with **one** asset attached, the console
196
- **reads** `change` while dispatch correctly falls back to **`change_vqa`**. This is not a bug: the
197
- reading describes the question's intent; the dispatch respects what can actually be computed with the
198
- assets present. It is documented so it is not mistaken for a defect.
 
 
 
 
 
 
 
199
 
200
  ---
201
 
202
- ## 7. Environment findings (would otherwise cost hours)
203
 
204
  | Finding | Detail |
205
  |---|---|
@@ -211,34 +541,218 @@ assets present. It is documented so it is not mistaken for a defect.
211
  | **Cloudflare `_headers` concatenate** | two matching rules are merged, not overridden; Chromium takes the **first** `max-age`. |
212
  | **Cloudflare 308-redirects `X.html` → `/X`** | reference the extensionless path. |
213
  | **A forwarded Codespace port returns `302`** | for a private repo — this is *why* the tunnel exists. |
214
- | **Chrome drops synthetic CDP key events without OS focus** | the harness false-positive (§4). |
215
  | **`browser-use` block-buffers stdout** | even when redirected; needs explicit line buffering to stream. |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
216
 
217
  ---
218
 
219
- ## 8. The BigEarthNet format contradiction (`ATTEMPTED`, reported not resolved)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
220
 
221
- The BigEarthNet data format **contradicts the original plan**. This was **reported rather than
222
- silently patched**, because quietly changing the preprocessing would move the frozen config hash.
 
 
 
223
 
224
- Two facts matter:
 
 
225
 
226
- 1. The BigEarthNet documentation — its uses, mentions, or endorsements — does **not** specify a
227
- percentile stretch. This project nevertheless applies percentile normalisation (2/98) for optical
228
- inputs to match the CROMA contract. That is a **deliberate, documented choice**, not an upstream
229
- fact.
230
- 2. The local subset is **100 % single-label**, against the official 1–11 multi-label scheme, so
231
- metrics computed on it are **not comparable** to published multi-label numbers.
232
 
233
  ---
234
 
235
- ## 9. Where the evidence lives
236
 
237
  | Topic | Evidence |
238
  |---|---|
239
- | Findings F4-1…F5-3, C-1…C-9, P7-1 | `docs/PHASE*.md`, `docs/ARCHITECTURE_*.md`, `docs/CROMA_NORMALISATION_UPSTREAM_EVIDENCE.md` |
 
 
 
 
240
  | The 448-vs-224 paired test | `docs/PHASE7_RESOLUTION_DECISION.md` |
241
  | The VLM rejection | `docs/PHASE6_RUN1_REJECTION_DIAGNOSIS.md`, `artifacts/vlm/phase6_closure.json` |
242
  | Router defect + 3 live passes | `.workbuddy-ai/scratch/live_validation/` (`run_output.txt`, `run_final2.txt`, `run_final3.txt`) |
243
  | The undeployed B-07 patch | session scratch `fix-b07-forward-unavailable.patch` |
244
  | Live validation harness | `.workbuddy-ai/scratch/run_all_postfix2.harness`, `recompute_verdicts.py` |
 
 
 
 
 
 
 
 
 
2
 
3
  Engineering findings, negative results, and design decisions that would otherwise be lost. Every one
4
  was learned by **probe or execution**, not assumption, and every one is recorded so it is not
5
+ rediscovered. Where a finding changed code, the change is named; where a finding was *reported rather
6
+ than patched*, the reason is given.
7
 
8
  **Status tags:** `MEASURED` · `RESOLVED` · `REJECTED` · `OPEN` · `ATTEMPTED`.
9
 
10
+ **How to read this document.** §2 lists the twelve numbered findings that changed the code
11
+ (F4-1…F5-3, P7-1, C-1, C-6…C-9), each with **what was measured**, **the consequence**, and **the file
12
+ that records it**. §3–§9 are the larger case studies: the pre-registered resolution rejection, the
13
+ router-defect fix, the harness false-positive, the `auto`-mode fallthrough, the
14
+ `interpret()`/`chooseTask()` asymmetry, the environment findings, and the BigEarthNet contradiction.
15
+
16
+ **Companions.** [`LIMITATIONS.md`](LIMITATIONS.md) (the exhaustive catalogue of what does not work),
17
+ [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md) (the environment traps in operator form),
18
+ [`architecture/04-router.md`](architecture/04-router.md), [`DEPLOYMENT.md`](DEPLOYMENT.md) §8.1.
19
+
20
+ ---
21
+
22
+ ## 1. The convention that matters most here
23
+
24
+ A finding is only recorded when it was **measured**, and the measurement is quoted rather than
25
+ summarised. Two examples of why this convention exists, both recorded in the source documents:
26
+
27
+ - A per-class guardrail (V2, below) was written with a 1.0 pp threshold that sits **below the
28
+ measurement resolution** of its own statistic — it flagged three classes whose drops were
29
+ statistically indistinguishable from zero. The rule was wrong, not the model.
30
+ - A "bounded, not fixed" classification of a path-disclosure finding was **wrong in three ways**, and
31
+ the third mattered most: it was not bounded and not measured. The recorded lesson is in the wording:
32
+ **`"bounded"` is a claim, and a claim is not a measurement.**
33
+
34
+ Where a value could not be established, the honest entry is *"not measured"* rather than a hedge.
35
+
36
  ---
37
 
38
+ ## 2. Findings that changed the code
39
+
40
+ ### F4-1 — the MiniLM tokenizer ceiling is 256, not 128 (`MEASURED`)
41
+
42
+ **What was measured.** The config said `router.max_length: 128`. A probe showed the MiniLM tokenizer's
43
+ **own ceiling is 256**. 128 is therefore a deliberate truncation *well inside* the ceiling, not the
44
+ model's limit — but the config read as if it were the latter.
45
+
46
+ **Consequence.** `router.max_length: 128` is retained, but `FrozenEncoder` now applies it explicitly
47
+ and **rejects** any value above 256, because truncating above the ceiling is a **silent no-op** — "a
48
+ control that appears to work and does nothing". Satellite queries are short, so halving the sequence
49
+ halves attention cost for no measurable accuracy loss.
50
+
51
+ **Recorded in.** `docs/PHASE4_ROUTER_REPORT.md` §F4-1; `docs/ARCHITECTURE_FREEZE.md`;
52
+ `artifacts/router/router_adapter_v001/metadata.json` (`encoder.max_length: 128`).
53
+
54
+ ### F4-2 — the router does not need a GPU (`MEASURED`)
55
+
56
+ **What was measured.** The encoder is frozen, so embeddings are a pure function of the query text.
57
+ Embedding the corpus measured **0.118 s / 64 queries on CPU**; training the **51,725-parameter** adapter
58
+ on the cached vectors measured **0.28 s for 20 epochs over 4,096 vectors**.
59
+
60
+ **Consequence.** The router can be retrained during development at **zero GPU quota cost**. The plan's
61
+ budget line — "Router | CPU/T4 | <1 h" — was roughly **three orders of magnitude pessimistic**; Phase 4
62
+ ran to completion locally.
63
+
64
+ > **Number discipline.** `artifacts/router/router_adapter_v001/metadata.json` records
65
+ > `num_parameters: 51725`. The measured figure is **51,725**; use it rather than any other recollection
66
+ > of the parameter count.
67
 
68
+ **Recorded in.** `docs/PHASE4_ROUTER_REPORT.md` §F4-2;
69
+ `artifacts/router/router_adapter_v001/metadata.json`.
70
 
71
+ ### F4-3 — the corpus needs *group*-level splitting (`MEASURED`)
 
 
 
72
 
73
+ **What was measured.** Template-generated queries are near-duplicates. Splitting by example would put
74
+ `"Show me the water body."` in train and `"Show me the road."` in val — one token apart — and report a
75
+ **fake** accuracy.
76
 
77
+ **Consequence.** Splits are by **group** (template id, or hard-negative family), never by example,
78
+ mirroring `evaluation.leakage.assign_splits_by_scene` deliberately: "the failure mode is identical, so
79
+ the guard should look identical". Hard-negative families are placed in the **test** split so their
80
+ accuracy measures generalisation rather than memorisation. The corpus metadata records
81
+ `groups: 54` over `total: 576` queries.
82
 
83
+ **Recorded in.** `docs/PHASE4_ROUTER_REPORT.md` §F4-3;
84
+ `artifacts/router/router_adapter_v001/metadata.json` (`corpus.groups`).
85
 
86
+ ### F5-1 — `AutoModelForVision2Seq` is absent, not deprecated (`MEASURED`)
87
 
88
+ **What was measured.** In transformers **5.17.0**:
89
 
90
+ | Class | Present? |
91
+ |---|---|
92
+ | `AutoModelForImageTextToText` | **present** |
93
+ | `AutoModelForVision2Seq` | **ABSENT** |
94
+ | `AutoModelForMultimodalLM` | present |
95
 
96
+ The class **does not exist** (it is not merely deprecated).
97
 
98
+ **Consequence.** The loader is resolved by **feature detection**, never hardcoded to one class name, so
99
+ a transformers version that renames or removes a class does not break the load path.
100
 
101
+ **Recorded in.** `docs/PHASE5_VLM_CONTRACT.md` §F5-1.
102
 
103
  ### F5-2 — the processor cost overrun is ~17×, not 4× (`MEASURED`)
104
 
105
+ **What was measured.** The processor's default `longest_edge` is **2048**, which upscales a 512-px tile
106
+ **4×** and then splits it (`do_image_splitting=True`) into sub-images:
107
 
108
+ ```
109
+ INPUT: one 512×512 RGB tile
110
+
111
+ DEFAULT (size.longest_edge = 2048, do_image_splitting = True)
112
+ pixel_values (1, 17, 3, 512, 512) <- 17 images
113
+ prompt tokens 1142
114
+
115
+ PINNED (processor_longest_edge = 512)
116
+ pixel_values (1, 1, 3, 512, 512) <- 1 image
117
+ ```
118
+
119
+ The plan estimated a 4× cost overrun; the real figure is **~17×**.
120
+
121
+ **Consequence.** `processor_longest_edge` **must** be set explicitly on the processor at construction
122
+ time, and `core/config.py` now **enforces** `processor_longest_edge <= image.tile_size` so the pin is a
123
+ control rather than a comment. Confirmed in the real load path: `max_images_seen = 1` — "the F5-2 pin
124
+ holds in the real load path, not just the probe. Unpinned it would read 17."
125
+
126
+ **Recorded in.** `docs/PHASE5_VLM_CONTRACT.md` §F5-2 (and the `max_images_seen` confirmation);
127
+ `docs/ARCHITECTURE_FREEZE.md` (`vlm.processor_longest_edge: 512`).
128
+
129
+ ### F5-3 — SmolVLM requires `<image>` tokens in the prompt (`MEASURED`)
130
 
131
+ **What was measured.** Hand-written prompt strings fail:
 
 
132
 
133
+ ```
134
+ ValueError: The total number of <image> tokens in the prompts should be the
135
+ same as the number of images passed. Found [0] <image> tokens and [1] images
136
+ ```
137
 
138
+ **Consequence.** Prompts are **always** built through `processor.apply_chat_template()`, enforced by
139
+ the config key `vlm.prompt_must_use_chat_template: true`.
140
 
141
+ **Recorded in.** `docs/PHASE5_VLM_CONTRACT.md` §F5-3.
 
142
 
143
+ ### P7-1 — the RemoteCLIP projected dimension is 512, not 768 (`MEASURED`)
144
 
145
+ **What was measured.** `visual.positional_embedding` is **768 wide**, but `visual.proj` is **(768, 512)**
146
+ — the embeddings comparable against the text tower are the **projected** ones, i.e. **512**
147
+ (`dim_match: True`). The grounding head's per-cell feature is therefore `4 × 512 = 2048`.
148
 
149
+ **Consequence.** Using 768 anywhere in the grounding path would be a **silent shape error**, caught only
150
+ at the similarity computation — after the patch features have already been computed and cached.
151
+ `grounding.encoder_projected_dim: 512` is declared in config so `core/config.py` can validate the head
152
+ **without importing torch**, and `RemoteCLIPEncoder._verify_contract()` asserts the same value against
153
+ the real model at load time.
154
+
155
+ **Recorded in.** `docs/PHASE7_GROUNDING_CONTRACT.md` §P7-1; `docs/ARCHITECTURE_FREEZE.md`.
156
 
157
  ### C-1 — the availability mask is consumed by the head, not by CROMA (`MEASURED`)
158
 
159
+ **What was measured.** CROMA always sees the canonical channel counts (**12 optical, 2 SAR**,
160
+ zero-filled to canonical order). The availability mask is applied by the **fusion head**, not the
161
+ encoder. The verified fusion-head input dimensions:
162
+
163
+ ```
164
+ optical_GAP (B, 768)
165
+ SAR_GAP (B, 768)
166
+ joint_GAP (B, 768)
167
+ optical_mask (B, 12) <- availability, from the sensor adapter
168
+ sar_mask (B, 2) <- availability, from the sensor adapter
169
+ ---------
170
+ concat (B, 2318)
171
+ ```
172
+
173
+ i.e. `input_dim = 3 × 768 + 12 + 2 = 2318`.
174
+
175
+ **Consequence.** Channel/band dropout during fusion-head training is **mandatory** — "it is what teaches
176
+ the head to trust the availability mask". The sensor adapter never fabricates a missing band; missing
177
+ channels are masked/zero-filled per the validated adapter policy.
178
+
179
+ **Recorded in.** `docs/ARCHITECTURE_FREEZE.md` §2.5 (C-1).
180
 
181
  ### C-6 — T4 is SM 7.5, so training uses fp16, not bf16 (`MEASURED`)
182
 
183
+ **What was measured.** The target GPU (Tesla T4) is compute capability **7.5**; bf16 tensor cores are
184
+ **unavailable** there.
185
+
186
+ **Consequence.** `training.precision: fp16`. The loader validates the value is one of `fp16|bf16|fp32`.
187
+ The fp16-vs-planned-bf16 deviation is documented as a finding in its own right, not hidden.
188
+
189
+ **Recorded in.** `docs/ARCHITECTURE_FREEZE.md` (`training.precision` row, C-6);
190
+ `docs/PHASE6_RUN1_REJECTION_DIAGNOSIS.md` §6.3.
191
 
192
  ### C-7 — `image_resolution % 8 == 0` (`MEASURED`)
193
 
194
+ **What was measured.** CROMA requires `image_resolution % 8 == 0`. The native value **120** satisfies it
195
+ and yields **225 patches**.
196
+
197
+ **Consequence.** Enforced at config load (`croma.image_resolution: 120`).
198
+
199
+ **Recorded in.** `docs/ARCHITECTURE_FREEZE.md` (`croma.image_resolution` row, C-7).
200
 
201
  ### C-8 — ZeroGPU does not support `torch.compile` (`MEASURED`)
202
 
203
+ **What was measured.** `torch.compile` must never be enabled on the (historical) ZeroGPU target.
204
+
205
+ **Consequence.** Enforced: the loader **fails startup** if `deployment.torch_compile` is true. This is
206
+ why setting `SATQUERY_TORCH_COMPILE=true` fails startup rather than silently taking effect (see
207
+ [`DEPLOYMENT.md`](DEPLOYMENT.md) §6.3).
208
+
209
+ **Recorded in.** `docs/ARCHITECTURE_FREEZE.md` (`deployment.torch_compile` row, C-8);
210
+ `configs/deploy.yaml` header.
211
 
212
  ### C-9 — STANet hyperparameters are upstream-verified (`MEASURED`)
213
 
214
+ **What was measured.** Change detection uses a STANet-style architecture with **upstream-verified**
215
+ hyperparameters: ResNet-18 encoder, **PAM** self-attention mode, tile **256** (non-overlapping),
216
+ threshold **0.50**, loss `0.5·BCE + 0.5·Dice`, `lr = 1e-3`, `batch_size = 8`. The LEVIR-CD split is
217
+ **7120 / 1024 / 2048** (256-px patches). Post-processing: threshold → morphological cleanup → connected
218
+ components → minimum-component filter.
219
+
220
+ **Consequence.** The implementation is **reimplemented, not vendored** — so the upstream hyperparameters
221
+ are pinned as the contract, and the trained head's embedded metadata carries the same values
222
+ (`sa_mode: "PAM"`, `width: 128`, `encoder: "resnet18"`).
223
+
224
+ **Recorded in.** `docs/ARCHITECTURE_FREEZE.md` §2.5 (C-9);
225
+ `artifacts/change/eval_test/eval_result.json` (`checkpoint_embedded_config`).
226
 
227
  ---
228
 
229
+ ## 3. The grounding resolution decision — a pre-registered rejection (`REJECTED`)
230
 
231
+ **Question.** Should grounding decode at **448** or **224**?
232
 
233
  **Answer: 224. 448 was rejected** — notable because the rejection was *pre-registered* and then
234
+ *confirmed* by a paired test over identical samples (n = 16,159), on a Tesla T4
235
+ (`--all --device cuda --tag full`).
236
+
237
+ ### 3.1 The rule, fixed before the result was seen
238
+
239
+ ```
240
+ 448 WINS if Recall@0.5 improves by >= 0.05 absolute
241
+ OR mean best IoU improves by >= 0.05 absolute
242
+ 224 WINS otherwise
243
+ INCONCLUSIVE if fewer than 30 samples were scored
244
+ ```
245
+
246
+ The artifact records `rule_changed_since_preregistration: false` — the rule was **not** modified after
247
+ the result was seen.
248
+
249
+ ### 3.2 The result
250
+
251
+ ```
252
+ 224 WINS
253
+ recall@0.5 gain 448/224 : -0.0022
254
+ bestIoU gain 448/224 : -0.0147
255
+ latency ratio : 1.59x
256
+ ```
257
+
258
+ Neither component came close to the +0.05 margin. Both were **negative**.
259
+
260
+ ### 3.3 Measured detail
261
+
262
+ | metric | 224 | 448 | delta |
263
+ |---|---|---|---|
264
+ | token grid | 7 × 7 = 49 | 14 × 14 = 196 | 4.0× tokens |
265
+ | attention cost (n²) | 1× | 16× | — |
266
+ | with boxes | 16159/16159 | 16159/16159 | — |
267
+ | **mean best IoU** | **0.0972** | 0.0825 | **−0.0147** |
268
+ | Recall@0.10 | **0.3298** | 0.2599 | **−0.0699** |
269
+ | Recall@0.25 | **0.1187** | 0.0944 | **−0.0243** |
270
+ | Recall@0.50 | **0.0234** | 0.0212 | **−0.0022** |
271
+ | matched IoU | 0.0972 | 0.0825 | −0.0147 |
272
+ | latency mean | **20.0 ms** | 31.8 ms | 1.59× |
273
+ | latency p90 | **20.9 ms** | 32.9 ms | 1.57× |
274
+ | peak VRAM | **592.1 MB** | 599.8 MB | +7.7 MB |
275
+ | wall time | **~8.5 min** | ~11.2 min | 1.32× |
276
+
277
+ **448 is worse on every quality metric and slower. There is no axis on which it wins.**
278
+
279
+ Best-IoU distribution — the shift is a whole-distribution move toward the zero-overlap bucket, not a
280
+ tail effect:
281
+
282
+ | bucket | 224 | 448 |
283
+ |---|---|---|
284
+ | 0.00–0.10 | 10,829 | 11,957 |
285
+ | 0.10–0.25 | 3,412 | 2,677 |
286
+ | 0.25–0.50 | 1,540 | 1,182 |
287
+ | 0.50–0.75 | 336 | 307 |
288
+ | 0.75–1.01 | 42 | 36 |
289
 
290
+ The 224 column dominates the top three buckets; 448 has ~1,100 more near-total misses.
291
+
292
+ ### 3.4 Paired analysis — independent confirmation
293
+
294
+ Both resolutions scored the **same 16,159 samples**, so the paired test removes between-object variance:
295
+
296
+ ```
297
+ paired samples : 16159
298
+ mean 224 : 0.0972
299
+ mean 448 : 0.0825
300
+ mean paired diff : -0.0147 (95% CI -0.0160 .. -0.0134)
301
+ t statistic : -22.63
302
+ CI excludes zero : True
303
+
304
+ 448 better on : 1371/16159 ( 8.5%)
305
+ 448 worse on : 3372/16159 (20.9%)
306
+ identical : 11416/16159 (70.6%)
307
+ ```
308
+
309
+ **The paired test and the pre-registered rule agree.** There is no rule-versus-evidence disagreement to
310
+ escalate: both say 224, and the confidence interval excludes zero by a wide margin. The win/loss split
311
+ is also informative: 448 wins on only **8.5 %** of records and loses on **20.9 %** — the finer grid is
312
+ not merely neutral, it is **actively harmful on a fifth of the corpus**.
313
+
314
+ ### 3.5 Recall ladder, paired
315
+
316
+ | threshold | 224 | 448 | diff | 95% CI |
317
+ |---|---|---|---|---|
318
+ | 0.10 | 0.3298 | 0.2599 | −0.0699 | excludes zero |
319
+ | 0.25 | 0.1187 | 0.0944 | −0.0243 | excludes zero |
320
+ | 0.50 | 0.0234 | 0.0212 | −0.0022 | excludes zero |
321
+
322
+ The gap **narrows as the threshold rises** — the signature of a method that cannot reach high IoU either
323
+ way. At IoU 0.50 the two are within 0.002 of each other and both are near the floor.
324
+
325
+ ### 3.6 Why 448 did not help — the honest reading
326
+
327
+ The zero-shot method selects a patch by text similarity and returns that patch's box. At 224 a box is
328
+ 1/7 of the image; at 448 it is 1/14. Two things work against the finer grid:
329
+
330
+ 1. **The peak is not sharper at 448.** Splitting each cell into four gives four chances to pick a wrong
331
+ sub-cell, and the similarity field on frozen features is smooth, so the argmax moves around. 448 wins
332
+ on 8.5 % and loses on 20.9 % — losses outnumber wins by 2.5 : 1.
333
+ 2. **Recall@0.10 drops the most (−0.0699).** If finer tokens genuinely localised better, the *loosest*
334
+ threshold would benefit most. It degrades most, which means the fine grid adds positional noise
335
+ rather than positional precision.
336
+
337
+ This is the **zero-shot baseline's** limitation, not a property of RemoteCLIP. A **learned** head
338
+ trained to regress boxes from these features may respond differently.
339
+
340
+ ### 3.7 What this establishes, and what it does not
341
+
342
+ Establishes: grounding runs at **224** (frozen in `configs/base.yaml`:
343
+ `grounding.image_size: 224`, `grounding.resolution_frozen: true`); peak VRAM for the frozen encoder at
344
+ 224 is **592 MB**; encoder latency at 224 on a T4 is **20 ms/image** (5× faster than the CPU figure of
345
+ 97 ms); the 224 localisation floor is **1/7 of image width per token**.
346
+
347
+ Does **not** establish: whether the zero-shot baseline is *good* (it is not — mean best IoU 0.0972 and
348
+ Recall@0.5 0.0234 are weak, and this is an ablation floor for the Phase-8 head, not a product); whether
349
+ a **trained** head has the same resolution sensitivity (re-opening the question after Phase 8 is
350
+ legitimate *if* the head's validation curve suggests it, and would be a **new** pre-registered
351
+ experiment, not a silent retune); anything about hidden ISRO/SAC imagery (VRSBench is overhead optical;
352
+ the hidden set is Cartosat-2S + RISAT, a different distribution entirely).
353
+
354
+ **Degeneracy note.** At n=12, n=40 and n=6 the smoke runs reported `Recall@0.5 = 0.0000` at **both**
355
+ resolutions and the script emitted a degeneracy warning; at full scale the metric is non-zero
356
+ (0.0234 / 0.0212), so the note correctly did not fire. The sub-floor runs were never treated as
357
+ evidence.
358
+
359
+ **Reproduction.**
360
+
361
+ ```bash
362
+ python scripts/exp_grounding_resolution.py \
363
+ --vrsbench <data-root> \
364
+ --checkpoint <RemoteCLIP-ViT-B-32.pt> \
365
+ --all --device cuda --tag full
366
+
367
+ python scripts/analyze_grounding_resolution.py --tag _full
368
+ ```
369
+
370
+ Artifacts: `per_sample_224_full.jsonl`, `per_sample_448_full.jsonl`,
371
+ `resolution_experiment_full.json` — 16,159 lines each. Every aggregate is recomputable from the JSONL
372
+ without re-running the encoder.
373
+
374
+ **Recorded in.** `docs/PHASE7_RESOLUTION_DECISION.md` (full document).
375
 
376
  ---
377
 
378
+ ## 4. The router defect — a real bug, found and fixed (`RESOLVED`)
379
 
380
+ ### 4.1 Symptom
381
 
382
+ The query *"Where are the built-up areas in this image?"* — with **one** asset attached — collapsed to
383
+ **`vqa`** and answered **"River"**, instead of routing to `grounding`. A second query, *"Where is the
384
+ new airport?"*, behaved the same way.
385
 
386
+ ### 4.2 Root cause
387
 
388
  Two functions with different information:
389
 
390
  - **`interpret()`** — produces the console's *reading*; **asset-count-blind** (text only).
391
  - **`chooseTask()`** — performs *dispatch*; **asset-count-aware**.
392
 
393
+ The defect was in the reading/dispatch path's handling of spatial/lexical cues. The pre-fix replay of
394
+ the shipped functions shows the mechanism exactly:
395
 
396
+ ```
397
+ "Where are the built-up areas in this image?" — one asset.
398
+ Before: reading=change, temporal=required → dispatched=vqa (wanted=change_vqa, substituted=true).
399
+ \bbuilt\b matched the temporal regex and `area` matched inside "areas".
400
+ ```
401
 
402
+ So the word "built" was in the temporal (change) set, and "area" matched as a substring of "areas".
403
+ Region queries fell through to the generic VQA specialist.
404
+
405
+ ### 4.3 Fix and verification
406
+
407
+ The fix was deployed to `SatQuery-Frontend` (`mission.js` `interpret()`, `ff46eba42b18`) and the sibling
408
+ `core.js` `SQ.policy` (`2d7ae53b482d`). It was validated by **three independent live passes**:
409
 
410
  | Pass | Deployed HEAD | Result |
411
  |---|---|---|
 
415
 
416
  Both defect queries now dispatch to `grounding`:
417
 
418
+ | Query | Pass-1 run id | Pass-2 run id | Pass-3 run id |
419
+ |---|---|---|---|
420
+ | Where are the built-up areas in this image? | `run_f0d7a90b5aa1` | `run_2a07dcdbae96` | **`run_467ffa406f22`** |
421
+ | Where is the new airport? | `run_69e38a182a71` | `run_9134f40a258c` | **`run_46980ba55c62`** |
422
+
423
+ **24 live runs, 24 correct dispatches, 0 mock nodes.** No run id is shared between passes. Post-fix the
424
+ live answers read `reading=grounding, temporal=none` and `[grounding] Located 6 candidate region(s) …
425
+ Highest objectness 0.82` / `0.83`. Screenshots (`A1…B2*.png`) are in the live-validation scratch
426
+ directory.
427
 
428
+ ### 4.4 The residuals the fix left (and did not hide)
429
+
430
+ - *"What is the new runway?"* still reads `change` rather than `vqa` (the `new`-as-change heuristic
431
+ fires on non-`where` questions). Strictly better than pre-fix, where `new` was unconditionally
432
+ temporal. "A lexical router cannot cleanly separate 'the new X' from 'what's new'."
433
+ - *"How much built-up area was added?"* now reads `vqa` (under-trigger), because `built` was dropped
434
+ from the temporal set and `area` no longer matches inside `areas`.
435
+
436
+ **Recorded in.** `docs/FINAL_DELIVERY_TODO.md` §5 B-08, §6 E-09/E-11/E-14;
437
+ `.workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md`; `run_final2.txt`, `run_final3.txt`;
438
+ `results_final.json`, `results_pass3.json`.
439
+
440
+ ### 4.5 The corpus the fix was validated against (and its limits)
441
+
442
+ The router is trained and validated on a **576-query** corpus in **54** groups
443
+ (`corpus_total: 576`, `corpus_groups: 54`), distributed by task: caption 91, change 115, grounding 128,
444
+ optical_sar 50, unsupported 105, vqa 87. The validation split used for the reported accuracy is
445
+ **n = 86**; the corpus is flagged `corpus_limited: true`. The adapter has **51,725** parameters over a
446
+ 384-dim frozen MiniLM encoder (6 task classes, 4 modalities, 3 binary heads).
447
+
448
+ This is why the router number must be read as **indicative only** — see
449
+ [`LIMITATIONS.md`](LIMITATIONS.md) §1 (L-12) and §8 (L-67). The fix was validated by **live behaviour**
450
+ (24 runs, 24 correct dispatches), not by a corpus accuracy jump, precisely because the corpus is small.
451
+
452
+ **Recorded in.** `artifacts/router/router_adapter_v001/metadata.json`;
453
+ `artifacts/router/threshold_sweep_val.json`.
454
 
455
  ---
456
 
457
+ ## 5. The harness false-positive — caught before it could lie (`RESOLVED`)
458
+
459
+ ### 5.1 What happened
460
 
461
  An earlier live-validation harness typed queries with **synthetic CDP key events**, which Chrome
462
+ **silently drops when the window lacks OS focus**. The harness therefore dispatched the page's *default*
463
+ query and still recorded a "result" — a **false pass**. Measured directly: with Chrome backgrounded,
464
+ `press_key("Z")` left `#qtext.value` unchanged, while `type_text("Q")` (CDP `Input.insertText`, not
465
+ focus-gated) inserted fine. The re-run attempt failed on case 1 with `run_id=0002`, `mock_nodes=9`,
466
+ `answer="No answer yet"`, and only a `capabilities` call — the **mock path**.
467
+
468
+ ### 5.2 Fix
469
 
470
+ The current harness **asserts form state before dispatch** (`q_ok`, `obs_ok`, `t0_ok`) and uses
471
+ deterministic query entry (`js()` value-set + `type_text()` via CDP `Input.insertText`). Two further
472
+ harness bugs were found and fixed, both causing **false failures**:
473
 
474
+ - the answer `[task]` tag exists only for region tasks (vqa/caption answers are bare); and
475
+ - the intent panel is a **concatenated** string, so `task([a-z_]+)` must be matched **non-greedily** up
476
+ to `modality`.
477
+
478
+ The harness now computes the dispatched task as `answer_tag` when present, else the reading.
479
+
480
+ ### 5.3 Independent check
481
+
482
+ The earlier 8/8 run was re-examined and confirmed **not** infected — its intents were query-specific
483
+ (e.g. A1 read `taskvqa…temporalnone`, not the default's `taskchange…temporalrequired`), its answers
484
+ embedded the query text, and A6's answer proved two files were uploaded. The failure mode is recorded
485
+ because it is exactly the **silent false-positive an evaluation harness must never have**.
486
+
487
+ **Recorded in.** `.workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md` ("Why this pass
488
+ needed a new harness"); `docs/FINAL_DELIVERY_TODO.md` §6 E-14.
489
 
490
  ---
491
 
492
+ ## 6. The `transport_mode: auto` fallthrough (`OPEN`)
493
 
494
  `SATQUERY_TRANSPORT=auto` tries the tunnel, then falls through to the forward path on timeout. The
495
  forward path to a **private** repo returns `302` quickly, but the wake step still consumes
496
  `SATQUERY_WAKE_TIMEOUT_S` (120 s) first — so a worst-case failed request takes ≈ **249 s**
497
+ (150 + 120). This is the **root shape** of the observed transient tunnel gap.
498
 
499
+ The recorded root cause is precise: in `auto` transport mode a tunnel timeout **falls through** to the
500
+ forward path (`SatQuery-Backend/main.py:546`), which then burns `wake_timeout_s=120` on a `302` → the
501
+ observed `504`.
502
+
503
+ A patch (`fix-b07-forward-unavailable.patch`) was authored and verified (`git apply --check` clean,
504
+ `py_compile` clean, applies to the deployed `89d80eaddec5`). It adds `forward_unavailable` (**503**,
505
+ terminal `302`/`401`/`403` on the forward path) and `upstream_timeout` (**504**, tunnel healthy but
506
+ slow) codes, plus the `codespace_name` `.strip()` fix.
507
+
508
+ > **Status: `OPEN`.** The patch is **prepared but NOT deployed.** The deployed health payload still
509
+ > shows the trailing `\n`.
510
+
511
+ **Recorded in.** `release/CURRENT_RELEASE_STATE.md` §6; `docs/FINAL_DELIVERY_TODO.md` §5 B-07;
512
+ [`DEPLOYMENT.md`](DEPLOYMENT.md) §8.1.
513
 
514
  ---
515
 
516
+ ## 7. The `interpret()` / `chooseTask()` asymmetry — intentional (`RESOLVED`)
517
 
518
  For *"What changed between the earlier and later image?"* with **one** asset attached, the console
519
+ **reads** `change` while dispatch correctly falls back to **`change_vqa`**. This is **not** a bug:
520
+
521
+ - the **reading** describes the question's intent (asset-count-blind);
522
+ - the **dispatch** respects what can actually be computed with the assets present (asset-count-aware).
523
+
524
+ It is documented so it is not mistaken for a defect. In the live-validation table this appears as the
525
+ one case where reading ≠ dispatched (`change` → `change_vqa`) — "flagged, not failed".
526
+
527
+ **Recorded in.** `.workbuddy-ai/scratch/live_validation/LIVE_VALIDATION_POSTFIX.md` ("Discriminator
528
+ note"); [`architecture/04-router.md`](architecture/04-router.md).
529
 
530
  ---
531
 
532
+ ## 8. Environment findings (would otherwise cost hours)
533
 
534
  | Finding | Detail |
535
  |---|---|
 
541
  | **Cloudflare `_headers` concatenate** | two matching rules are merged, not overridden; Chromium takes the **first** `max-age`. |
542
  | **Cloudflare 308-redirects `X.html` → `/X`** | reference the extensionless path. |
543
  | **A forwarded Codespace port returns `302`** | for a private repo — this is *why* the tunnel exists. |
544
+ | **Chrome drops synthetic CDP key events without OS focus** | the harness false-positive (§5). |
545
  | **`browser-use` block-buffers stdout** | even when redirected; needs explicit line buffering to stream. |
546
+ | **WDAC blocked `orjson`, so FastAPI could not import** *(historical — no longer reproduces)* | `ImportError: DLL load failed while importing orjson: An Application Control policy has blocked this file.` Verified as `OSError [WinError 4551]` from a direct `ctypes.CDLL` on the binary. FastAPI's own guard catches `ModuleNotFoundError`, not a policy-blocked `ImportError`, so it aborted the import. When the block lifted, **four live defects became reachable** that had been sitting in `gateway/` the whole time (G-1…G-4). Do not cite this as a current limitation. |
547
+ | **`pandas._libs.parsers` / `sparsefuncs_fast` App Control block** | 56 evaluation-side tests remain uncollectable. Unchanged; unrelated to the backend chain. |
548
+ | **CPU-only torch** | CUDA autocast is a no-op; no GPU path was exercised in the authoring environment. |
549
+
550
+ ### 8.1 A withdrawn explanation, recorded rather than deleted
551
+
552
+ The reason that nine backend-chain tests (D–I, Q, R) were NOT RUN was originally recorded as "the
553
+ environment's egress proxy intercepts outbound HTTP and returns `502`". That explanation was
554
+ **re-measured on 2026-09-22 and does not hold**: `https://huggingface.co` returned **HTTP 200**,
555
+ `/api/models` returned 200 with real JSON, and a real weight-file path returned 200. The correct reason
556
+ is that **no upstream exists** — no Space and no service has ever been deployed, and the gateway is a
557
+ pure proxy that holds no second copy of the capability table, so a metadata request *is* an upstream
558
+ request. The classification (NOT RUN / ENVIRONMENT-BLOCKED) is unchanged; only the reason changed.
559
+
560
+ ### 8.2 The transferable lesson from the stale-negative class
561
+
562
+ Three documents written after the Phase-12 A/B experiment still described it as un-run, even though it
563
+ had completed 34 hours earlier. **A stale negative claim is more dangerous than a stale positive one**:
564
+ a wrong number is eventually contradicted by the artifact it describes, but "this has never been
565
+ executed" is contradicted by *nothing* — no test fails, no hash moves, no invariant breaks. The rule
566
+ this suggests: **never assert that an artifact does not exist from memory; assert it from a command, and
567
+ record the command.** No test can check that a documented absence is still absent.
568
+
569
+ **Recorded in.** `docs/PHASE19_FINAL_HARDENING.md` §5 (historical blocker) and §5.3 (CPU-only torch);
570
+ `docs/STEP7_BACKEND_CHAIN_REPORT.md` §5, §11, §13, §15; `docs/PHASE12_CURRENCY_CORRECTION.md` §6;
571
+ [`REPRODUCIBILITY.md`](REPRODUCIBILITY.md).
572
+
573
+ ---
574
+
575
+ ## 9. The BigEarthNet format contradiction (`ATTEMPTED`, reported not resolved)
576
+
577
+ The BigEarthNet data format **contradicts the original plan**. This was **reported rather than silently
578
+ patched**, because quietly changing the preprocessing would move the frozen config hash. Two facts
579
+ matter:
580
+
581
+ 1. **The percentile stretch is not upstream.** The BigEarthNet documentation — its uses, mentions, or
582
+ endorsements — does **not** specify a percentile stretch. This project nevertheless applies
583
+ percentile normalisation (**2/98**) for optical inputs to match the CROMA contract. That is a
584
+ **deliberate, documented choice**, not an upstream fact.
585
+ 2. **The local subset is single-label.** It is **100 % single-label** against the official **1–11
586
+ multi-label** scheme, so metrics computed on it are **not comparable** to published multi-label
587
+ numbers.
588
+
589
+ **Consequence.** The contradiction is recorded as a limitation and an open item rather than resolved by
590
+ editing preprocessing (which would move `Config.hash` off `78f1e3700da15aa1`).
591
+
592
+ **Recorded in.** [`LIMITATIONS.md`](LIMITATIONS.md) §3 (L-24);
593
+ `docs/PHASE12_LABEL_POLICY_DECISION.md`; `docs/PHASE14_CROMA_NORMALISATION_CHANGE.md`.
594
 
595
  ---
596
 
597
+ ## 10. Additional measured defects that shaped the design
598
+
599
+ These are not headline findings, but each changed a decision or a guard. They are recorded together
600
+ because they share a shape: a value or a claim that was produced without a measurement, then corrected
601
+ by one.
602
+
603
+ ### 10.1 The train/serve skew (the F2 finding) (`RESOLVED`)
604
+
605
+ **What was measured.** `scripts/prepare_change_vqa.py` builds its change features from the **trained**
606
+ STANet (`DEFAULT_CHANGE_CHECKPOINT`), but serving had no equivalent wiring: `change.checkpoint_path` is
607
+ unset in `configs/base.yaml`, so the registry passed no `checkpoint_path` and the change-VQA specialist
608
+ would construct an **untrained** STANet and answer from a representation the head was never fitted on.
609
+
610
+ **Consequence.** `app/serving.py::_wired_change_vqa_builder` supplies the **same** checkpoint the
611
+ change builder uses, so the detector behind a change answer and the detector behind the head's training
612
+ features are one artifact by construction. The specialist's own `feature_spec_mismatch()` check stays
613
+ armed as a second line of defence, not the only one.
614
+
615
+ **Recorded in.** `app/serving.py` (`_wired_change_vqa_builder` docstring).
616
+
617
+ ### 10.2 The optical-SAR encoder was unreachable by default (`RESOLVED`)
618
+
619
+ **What was measured.** `specialists/optical_sar/specialist.py` builds CROMA only when handed a
620
+ `checkpoint_path` that exists. `croma.checkpoint_path` is not in `configs/base.yaml` — and must not be,
621
+ or `Config.hash` moves — so the registry passed no path, the gate was False, and the default serving
622
+ composition ran with `encoder=None` while the checkpoint sat on disk the whole time. The registry then
623
+ correctly reported `DEGRADED` ("no encoder; running on fallback"): a deployment that could never answer
624
+ an optical/SAR question.
625
+
626
+ **Consequence.** `app/serving.py::_wired_optical_sar_builder` resolves the checkpoint from the **pinned
627
+ identity** through the hash-exempt channel (`croma.resolve_checkpoint_path`: env → config → pinned Hub
628
+ cache, offline first) and wires the fusion head the same way. The resolution `source` is logged so
629
+ "which checkpoint did this process actually use" is answerable from the trace.
630
+
631
+ **Recorded in.** `app/serving.py` (`_wired_optical_sar_builder` docstring);
632
+ `specialists/optical_sar/specialist.py`.
633
+
634
+ ### 10.3 G-1 … G-4 — four defects the first real ASGI run found (`RESOLVED`)
635
+
636
+ **What was measured.** The ASGI layer had **never executed** in the repository, and the reason was
637
+ recorded in the code as an environment blocker (a WDAC block on `orjson.pyd`). When the block lifted,
638
+ within the first hour of actually running the server **four real defects surfaced**, four of them latent
639
+ for as long as the blocker was believed:
640
+
641
+ | ID | Defect | Severity |
642
+ |---|---|---|
643
+ | **G-1** | `request: Request` never resolved — an in-function import left `Request` out of `__globals__`, so FastAPI silently reinterpreted the parameter as a required **query** parameter named `request`; **every POST body was misread** and no handler ran | **Critical** |
644
+ | **G-2** | an unsupported `force_task` enum value was forwarded upstream instead of refused locally → wrong status, wasted round trip | High |
645
+ | **G-3** | an empty `HF_TOKEN` produced `Authorization: Bearer `, which httpx rejects → a crash reported as an upstream failure | High |
646
+ | **G-4** | a non-JSON upstream error body was relayed verbatim → contract break and internal-text disclosure | High |
647
+
648
+ **Consequence.** All four fixed, with regression tests that assert the **cause** rather than the
649
+ symptom. The transferable lesson: "a documented blocker is a place where evidence stops, and nothing was
650
+ watching for the blocker to lift."
651
+
652
+ **Recorded in.** `docs/STEP7_BACKEND_CHAIN_REPORT.md` §13, §14.
653
+
654
+ ### 10.4 F-1 … F-7 — the gateway layer-crossing audit (`RESOLVED` / recorded)
655
+
656
+ **What was measured.** A series of findings, each found by driving the real ASGI stack rather than by
657
+ reading:
658
+
659
+ | ID | Finding | Disposition |
660
+ |---|---|---|
661
+ | **F-1** | a vacuous assertion in the gateway's own test suite | fixed |
662
+ | **F-2** | an upstream CORS header bypassed the allowlist entirely (**security**) | fixed |
663
+ | **F-3** | `404`/`405` did not carry the contract's error envelope | fixed (gateway); `app/space_app.py` does not register the handler (bounded defect F-12b) |
664
+ | **F-4** | `rate_limited` was emitted but documented nowhere | fixed |
665
+ | **F-5** | the per-IP rate limiter is defeated by a client-supplied `X-Forwarded-For` (measured 5/8 throttled without the header, **0/8** with a fresh value per request) | **ruled**: fairness only, not a security control |
666
+ | **F-6** | the body-size cap was **declarative**, not enforced — an omitted `Content-Length` buffered the entire body past the cap (measured: 12 MiB body → 13.9 MiB peak; allocation tracked body size with no ceiling) | fixed with a streaming check alongside the header check |
667
+ | **F-7** | two parsers of one variable (`SATQUERY_MAX_FILE_BYTES`) diverged in opposite directions | fixed; both layers now refuse a malformed value and name the variable |
668
+
669
+ **Consequence.** The rate limiter's claim was narrowed to what the measurement supports, and the size
670
+ cap is now enforced twice — deliberately, because "the header check protects the gateway's memory
671
+ against honest clients, not against hostile ones".
672
+
673
+ **Recorded in.** `docs/STEP8_FINAL_CONFORMANCE_AUDIT.md` §15–§17;
674
+ `docs/DEPLOYMENT_ARCHITECTURE.md` §5.2, §4 (F-6 note).
675
+
676
+ ### 10.5 F-13 … F-17 — what an unauthenticated client could learn (`RESOLVED` / recorded)
677
+
678
+ **What was measured.** A pass-by-pass audit of every value written to a client-visible field found that
679
+ `trace.inputs` (F-13), `trace.steps[PARSE].detail["inputs"]` (F-14), and the construction-failure
680
+ exception string (F-15) published **server-side filesystem paths** to an unauthenticated client —
681
+ F-15's case was "strictly worse" because the path was purely server-side and reachable on the **first**
682
+ request. F-16 found `result.change_map` / `evidence[].artifact_ref` carrying a real filesystem path
683
+ where the contract promised an `artifact://` URI. F-17 found `change_vqa.artifact_dir` configured but
684
+ never read.
685
+
686
+ **Consequence.** The owner ruled: sanitize client-facing exception messages to a **basename** (not a
687
+ replacement — the reason survives) and log the raw detail server-side; set an unavailable `artifact_ref`
688
+ to `null` with an explicit non-retrievable warning; do **not** fabricate `artifact://` URIs. The F-15
689
+ ruling needed **four** carriers, not three, because `result.execution_trace = trace` serializes the
690
+ trace object **twice**. F-17 is **documented, not patched**.
691
+
692
+ **Recorded in.** `docs/DEPLOYMENT_ARCHITECTURE.md` §5.3–§5.6; [`SECURITY.md`](SECURITY.md).
693
+
694
+ ### 10.6 The single capability authority (`RESOLVED`)
695
+
696
+ **What was measured.** Two independent producers answered "what can this deployment do?":
697
+ `AnalysisController.health()` reported **6** capabilities from the registry; `space_app.describe_deployment()`
698
+ reported **2** from two `Path.exists()` calls — and only the latter was served. Also measured:
699
+ `HealthStatus(**controller.health())` **fails** with 3 `extra_forbidden` errors.
700
+
701
+ **Consequence.** Owner ruling: the **registry is authoritative**, and `app/deployment.py` is the single
702
+ adapter that derives contract vocabulary from the registry's spec table and the filesystem. This is the
703
+ "three-copy problem": any table that exists in two places will drift, and this one already had.
704
+
705
+ **Recorded in.** `docs/DEPLOYMENT_ARCHITECTURE.md` §3.3.1;
706
+ `docs/STEP7_BACKEND_CHAIN_REPORT.md` §16.
707
+
708
+ ### 10.7 T-1 / T-2 — two test-side findings worth carrying (`RESOLVED`)
709
+
710
+ **What was measured.** (T-1) An earlier note recorded "22 documented error codes"; the real taxonomy and
711
+ the contract both have **23**. The test now parses the contract's §5.2 table and compares it against
712
+ `core/errors.py`, so the two must agree. (T-2) Three documentation tests initially failed by matching the
713
+ **prohibition itself** (e.g. a search for `gr.Blocks` matched the docstring that forbids it); the tests
714
+ now parse the module with `ast` and strip docstrings before searching — "a search for a forbidden token
715
+ must run over executable code".
716
+
717
+ **Consequence.** Both corrections became **bidirectional** checks rather than one-off fixes.
718
+
719
+ **Recorded in.** `docs/STEP7_BACKEND_CHAIN_REPORT.md` §3, §13; `docs/PHASE19_FINAL_HARDENING.md` §3.7.
720
+
721
+ ### 10.8 The calibration measurement, stated exactly (`MEASURED`)
722
 
723
+ **What was measured.** Temperature scaling was fitted on the Val split: `temperature = 0.9772731820958189`,
724
+ `n_samples = 16441`, `ece_before = 0.013755`, `ece_after = 0.014929`,
725
+ `ece_improvement = −0.001174` (**worse**), `nll_before = 0.6897411`, `nll_after = 0.6896308`,
726
+ `nll_improvement = +0.0001104`, `hit_bound = false`, `effective = true`. The raw softmax was **already
727
+ near-calibrated**, and temperature scaling made ECE very slightly worse while improving NLL marginally.
728
 
729
+ **Consequence.** The path is live (`core/controller.py` → `EvidenceEngine`), so a deployed result
730
+ carries a calibrated value; but this is a **measurement, not a quality judgment**, and must not be
731
+ described as scaling being "more accurate".
732
 
733
+ **Recorded in.** `artifacts/calibration_v001.json`; `docs/STEP7_BACKEND_CHAIN_REPORT.md` §7.
 
 
 
 
 
734
 
735
  ---
736
 
737
+ ## 11. Where the evidence lives
738
 
739
  | Topic | Evidence |
740
  |---|---|
741
+ | Findings F4-1…F4-3 | `docs/PHASE4_ROUTER_REPORT.md` |
742
+ | Findings F5-1…F5-3 | `docs/PHASE5_VLM_CONTRACT.md` |
743
+ | Finding P7-1 | `docs/PHASE7_GROUNDING_CONTRACT.md` |
744
+ | Findings C-1, C-6…C-9 | `docs/ARCHITECTURE_FREEZE.md` |
745
+ | CROMA normalisation upstream evidence | `docs/CROMA_NORMALISATION_UPSTREAM_EVIDENCE.md` |
746
  | The 448-vs-224 paired test | `docs/PHASE7_RESOLUTION_DECISION.md` |
747
  | The VLM rejection | `docs/PHASE6_RUN1_REJECTION_DIAGNOSIS.md`, `artifacts/vlm/phase6_closure.json` |
748
  | Router defect + 3 live passes | `.workbuddy-ai/scratch/live_validation/` (`run_output.txt`, `run_final2.txt`, `run_final3.txt`) |
749
  | The undeployed B-07 patch | session scratch `fix-b07-forward-unavailable.patch` |
750
  | Live validation harness | `.workbuddy-ai/scratch/run_all_postfix2.harness`, `recompute_verdicts.py` |
751
+ | The withdrawn egress explanation | `docs/STEP7_BACKEND_CHAIN_REPORT.md` §5 |
752
+ | The stale-negative lesson | `docs/PHASE12_CURRENCY_CORRECTION.md` §6 |
753
+ | The BigEarthNet contradiction | `docs/PHASE12_LABEL_POLICY_DECISION.md`, `docs/PHASE14_CROMA_NORMALISATION_CHANGE.md` |
754
+ | The gateway layer-crossing findings (F-1…F-7) | `docs/STEP8_FINAL_CONFORMANCE_AUDIT.md` §15–§17 |
755
+ | The ASGI defects (G-1…G-4) | `docs/STEP7_BACKEND_CHAIN_REPORT.md` §13 |
756
+ | The path-disclosure rulings (F-13…F-17) | `docs/DEPLOYMENT_ARCHITECTURE.md` §5.3–§5.6 |
757
+ | The train/serve skew and optical-SAR wiring | `app/serving.py` |
758
+ | The calibration measurement | `artifacts/calibration_v001.json`, `docs/STEP7_BACKEND_CHAIN_REPORT.md` §7 |