release: add docs/RESEARCH_NOTES.md

docs/RESEARCH_NOTES.md

@@ -1,7 +1,8 @@
 # Research Notes
 
-Engineering findings, negative results and design decisions that would otherwise be lost. Each was
-learned by **probe or execution**, not by assumption, and each is recorded so it is not rediscovered.
+Engineering findings, negative results, and design decisions that would otherwise be lost. Every one
+was learned by **probe or execution**, not assumption, and every one is recorded so it is not
+rediscovered.
 
 **Status tags:** `MEASURED` · `RESOLVED` · `REJECTED` · `OPEN` · `ATTEMPTED`.
 
@@ -11,32 +12,37 @@ learned by **probe or execution**, not by assumption, and each is recorded so it
 
 ### F4-1 — the MiniLM tokenizer ceiling (`MEASURED`)
 
-The MiniLM tokenizer's own ceiling is **256** (verified by probe). The project truncates to **128** —
+The MiniLM tokenizer's own ceiling is **256** (verified by probe). This project truncates to **128** —
 a deliberate truncation *well inside* the ceiling, not the model limit. Satellite queries are short;
 halving the sequence halves attention cost for no measurable accuracy loss. The encoder **asserts**
-`max_length ≤ 256`, because truncating above the ceiling is a silent no-op.
+`max_length ≤ 256`, because truncating above the ceiling is a **silent no-op**.
+
+**Consequence:** `router.max_length: 128` with an assertion, rather than a comment.
 
 ### F4-2 — the router needs no GPU (`MEASURED`)
 
 The encoder is frozen, so embeddings are **cached** and the 50,822-parameter adapter trains on cached
-vectors. **Measured on CPU: 20 epochs / 4,096 vectors in 0.28 s.** No GPU required.
+vectors. **Measured on CPU: 20 epochs / 4,096 vectors in 0.28 s.**
+
+**Consequence:** the router can be retrained during development without a GPU.
 
 ### F4-3 — splits must be by group (`MEASURED`)
 
 Splits are by **group** (template / hard-negative family), never by example. Hard-negative families
 are placed in the **test** split so their accuracy measures generalisation rather than memorisation.
-Splitting by example would leak template variants across the boundary.
+Splitting by example would leak template variants across the boundary and inflate the score.
 
 ### F5-1 — `AutoModelForVision2Seq` does not exist (`MEASURED`)
 
-In transformers 5.17.0, `AutoModelForVision2Seq` **does not exist** (not merely deprecated);
-`AutoModelForImageTextToText` is present. The loader is resolved by **feature detection**, never
-hardcoded.
+In transformers 5.17.0, `AutoModelForVision2Seq` **does not exist** (it is not merely deprecated);
+`AutoModelForImageTextToText` is present.
+
+**Consequence:** the loader is resolved by **feature detection**, never hardcoded to one class name.
 
 ### F5-2 — the processor cost overrun is ~17×, not 4× (`MEASURED`)
 
-The processor's default `longest_edge` is **2048**, which upscales 512-px tiles **4×** and then splits
-them (`do_image_splitting=True`) into **17 sub-images**:
+The processor's default `longest_edge` is **2048**, which upscales a 512-px tile **4×** and then
+splits it (`do_image_splitting=True`) into sub-images:
 
 | Setting | `pixel_values` | prompt tokens |
 |---|---|---|
@@ -44,18 +50,25 @@ them (`do_image_splitting=True`) into **17 sub-images**:
 | pinned (`processor_longest_edge: 512`) | `(1, 1, 3, 512, 512)` | — |
 
 The plan estimated a 4× cost overrun; the real figure is **~17×**. The value **must** be set
-explicitly on the processor at construction time.
+explicitly on the processor at construction time, and `core/config.py` now **enforces**
+`processor_longest_edge <= image.tile_size` so the pin is a control rather than a comment.
 
 ### F5-3 — prompts must go through the chat template (`MEASURED`)
 
 SmolVLM requires one `<image>` token per image in the prompt. Hand-written prompt strings raise
-`ValueError`. Prompts are always built through `processor.apply_chat_template()`.
+`ValueError`.
+
+**Consequence:** prompts are always built through `processor.apply_chat_template()`, enforced by the
+config key `vlm.prompt_must_use_chat_template: true`.
 
 ### P7-1 — the RemoteCLIP projected dimension is 512 (`MEASURED`)
 
 The RemoteCLIP ViT-B/32 transformer width is **768**, but `visual.proj` maps to a **projected** dim of
-**512**. The grounding head's per-cell feature is `4 × 512 = 2048`, declared in config so
-`core/config.py` can validate the head **without importing torch**.
+**512**. The grounding head's per-cell feature is therefore `4 × 512 = 2048`.
+
+**Consequence:** `grounding.encoder_projected_dim: 512` is declared in config so `core/config.py` can
+validate the head **without importing torch**, and `specialists/grounding/remoteclip.py` asserts the
+same value against the real model at load time.
 
 ### C-1 — the availability mask is consumed by the head, not by CROMA (`MEASURED`)
 
@@ -64,21 +77,23 @@ by the **fusion head** (`input_dim = 3 × 768 + 12 + 2 = 2318`), not by the enco
 
 ### C-6 — T4 is SM 7.5, so training uses fp16, not bf16 (`MEASURED`)
 
-The training precision is **fp16** because the target GPU (T4) is compute capability 7.5. bf16 is
-not available there.
+`training.precision: fp16` because the target GPU (T4) is compute capability 7.5; bf16 is unavailable
+there. The loader validates the value is one of `fp16|bf16|fp32`.
 
 ### C-7 — `image_resolution % 8 == 0` (`MEASURED`)
 
-CROMA requires `image_resolution % 8 == 0`. The native value **120** yields 225 patches.
+CROMA requires `image_resolution % 8 == 0`. The native value **120** yields 225 patches. Enforced at
+config load.
 
 ### C-8 — ZeroGPU does not support `torch.compile` (`MEASURED`)
 
-`torch.compile` must never be enabled on the (historical) ZeroGPU target.
+`torch.compile` must never be enabled on the (historical) ZeroGPU target. Enforced: the loader
+**fails startup** if `deployment.torch_compile` is true.
 
-### C-9 — STANet change hyperparameters are upstream-verified (`MEASURED`)
+### C-9 — STANet hyperparameters are upstream-verified (`MEASURED`)
 
-Change detection uses STANet-style architecture with upstream-verified hyperparameters (PAM
-self-attention, ResNet-18 encoder).
+Change detection uses a STANet-style architecture with upstream-verified hyperparameters: ResNet-18
+encoder, **PAM** self-attention mode, tile 256, threshold 0.50, BCE 0.5 + Dice 0.5.
 
 ---
 
@@ -86,27 +101,28 @@ self-attention, ResNet-18 encoder).
 
 **Question:** should grounding decode at 448 or 224?
 
-**Answer: 224. 448 was rejected** — and the rejection is notable because it was *pre-registered* and
-then *confirmed* by a paired test:
+**Answer: 224. 448 was rejected** — notable because the rejection was *pre-registered* and then
+*confirmed* by a paired test over identical samples (n = 16,159):
 
-| Comparison (over identical samples, n = 16,159) | 448 vs 224 |
+| Comparison (448 vs 224) | Value |
 |---|---|
 | mean best IoU | **−0.0147** |
 | recall@0.5 | −0.0022 |
 | recall@0.10 | −0.0699 |
 | recall@0.25 | −0.0243 |
 | latency | **1.59×** |
-| paired 95 % CI | [−0.0160, −0.0134] |
+| paired mean difference | −0.0147 |
+| paired 95 % CI | **[−0.0160, −0.0134]** |
 | paired t | **−22.63** |
 | 448 better on | 8.5 % of records |
 | 448 worse on | **20.9 %** of records |
 
 448 lost on **every** axis. The pre-registered decision rule and the paired test **agree** on 224.
-This is a model of how a resolution decision should be made: declared in advance, then tested.
+This is the model for how a resolution decision should be made: declared in advance, then tested.
 
 ---
 
-## 3. The router defect — a real bug, found and fixed
+## 3. The router defect — a real bug, found and fixed (`RESOLVED`)
 
 ### 3.1 Symptom
 
@@ -121,10 +137,10 @@ Two functions with different information:
 - **`interpret()`** — produces the console's *reading*; **asset-count-blind** (text only).
 - **`chooseTask()`** — performs *dispatch*; **asset-count-aware**.
 
-The defect was in the dispatch path's handling of spatial/lexical cues, so region queries fell
-through to the generic VQA specialist. See [`ARCHITECTURE.md`](ARCHITECTURE.md) §4.
+The defect was in the dispatch path's handling of spatial/lexical cues, so region queries fell through
+to the generic VQA specialist.
 
-### 3.3 Fix and verification (`RESOLVED`)
+### 3.3 Fix and verification
 
 The fix was deployed to `SatQuery-Frontend` and validated by **three independent live passes**:
 
@@ -141,18 +157,18 @@ Both defect queries now dispatch to `grounding`:
 | Where are the built-up areas in this image? | `run_467ffa406f22` | `grounding` |
 | Where is the new airport? | `run_46980ba55c62` | `grounding` |
 
-24 live runs, 24 correct dispatches, **0 mock nodes**. Screenshots are in
+**24 live runs, 24 correct dispatches, 0 mock nodes.** Screenshots are in
 [`../screenshots/`](../screenshots/).
 
 ---
 
-## 4. The harness false-positive — caught before it could lie
+## 4. The harness false-positive — caught before it could lie (`RESOLVED`)
 
 An earlier live-validation harness typed queries with **synthetic CDP key events**, which Chrome
 **silently drops when the window lacks OS focus**. The harness therefore dispatched the page's
 *default* query and still recorded a "result" — a **false pass**.
 
-**Fix:** the current harness **asserts form state before dispatch** (`q_ok`, `obs_ok`, `t0_ok`), and
+**Fix:** the current harness **asserts form state before dispatch** (`q_ok`, `obs_ok`, `t0_ok`) and
 uses deterministic query entry (`js()` value-set + `type_text()` via CDP `Input.insertText`).
 
 **Independent check:** the earlier 8/8 run was re-examined and confirmed **not** infected — its
@@ -165,9 +181,12 @@ recorded because it is exactly the silent false-positive an evaluation harness m
 
 `SATQUERY_TRANSPORT=auto` tries the tunnel, then falls through to the forward path on timeout. The
 forward path to a **private** repo returns `302` quickly, but the wake step still consumes
-`SATQUERY_WAKE_TIMEOUT_S` (120 s) first — so a worst-case failed request takes ≈ 249 s
-(150 + 120). This is the root shape of the observed transient tunnel gap. Recorded as **OPEN**; a
-deployed fix for the `codespace_name` newline on the wake path was authored separately.
+`SATQUERY_WAKE_TIMEOUT_S` (120 s) first — so a worst-case failed request takes ≈ **249 s**
+(150 + 120). This is the root shape of the observed transient tunnel gap.
+
+A patch adding `forward_unavailable` (503) and `upstream_timeout` (504) codes plus the
+`codespace_name` `.strip()` fix was authored and verified (`py_compile` clean). **Status: OPEN — the
+patch is prepared but NOT deployed.**
 
 ---
 
@@ -175,7 +194,7 @@ deployed fix for the `codespace_name` newline on the wake path was authored sepa
 
 For *"What changed between the earlier and later image?"* with **one** asset attached, the console
 **reads** `change` while dispatch correctly falls back to **`change_vqa`**. This is not a bug: the
-reading describes the question's intent, the dispatch respects what can actually be computed with the
+reading describes the question's intent; the dispatch respects what can actually be computed with the
 assets present. It is documented so it is not mistaken for a defect.
 
 ---
@@ -184,21 +203,42 @@ assets present. It is documented so it is not mistaken for a defect.
 
 | Finding | Detail |
 |---|---|
-| **Dead proxy in the sandbox** | outbound calls need `--noproxy '*'` (curl) or `ProxyHandler({})` (Python). |
+| **Dead proxy in the authoring sandbox** | outbound calls need `--noproxy '*'` (curl) or `ProxyHandler({})` (Python). |
+| **The sandbox proxy is slow for uploads** | `huggingface_hub` uploads stalled at ~51 kB/s through `http_proxy=127.0.0.1:58294`; the fix is to unset `http_proxy`/`https_proxy` and set `no_proxy='*'`. |
+| **`hf_hub_download` returned an EMPTY file** | sha256 `e3b0c442…` (the empty-content hash), which produced a **false FAIL** for all six artifacts. The honest check is a direct HTTPS download with `ProxyHandler({})`. |
 | **pytest is only in the repo venv** | `.venv/Scripts/python.exe`; a bare `pytest` misses it. |
-| **Full-suite pytest trips a bulk-delete guard** | sandbox-specific; affects `test_safe_delete_shim`. |
+| **The full test suite trips a bulk-delete guard** | sandbox-specific; affects `test_safe_delete_shim`. |
 | **Cloudflare `_headers` concatenate** | two matching rules are merged, not overridden; Chromium takes the **first** `max-age`. |
 | **Cloudflare 308-redirects `X.html` → `/X`** | reference the extensionless path. |
-| **A forwarded Codespace port returns `302`** for a private repo | this is *why* the tunnel exists. |
+| **A forwarded Codespace port returns `302`** | for a private repo — this is *why* the tunnel exists. |
 | **Chrome drops synthetic CDP key events without OS focus** | the harness false-positive (§4). |
-| **`browser-use` block-buffers stdout** even when redirected | needs explicit line buffering to stream. |
+| **`browser-use` block-buffers stdout** | even when redirected; needs explicit line buffering to stream. |
 
 ---
 
 ## 8. The BigEarthNet format contradiction (`ATTEMPTED`, reported not resolved)
 
 The BigEarthNet data format **contradicts the original plan**. This was **reported rather than
-silently patched**, because quietly changing the preprocessing would move the frozen config hash. The
-BigEarthNet documentation does not specify a percentile stretch; this project nevertheless applies
-percentile normalisation (2/98) to match the CROMA contract. That is a **deliberate, documented
-choice**, not an upstream fact. See [`DATASETS.md`](DATASETS.md) §5.3.
+silently patched**, because quietly changing the preprocessing would move the frozen config hash.
+
+Two facts matter:
+
+1. The BigEarthNet documentation — its uses, mentions, or endorsements — does **not** specify a
+   percentile stretch. This project nevertheless applies percentile normalisation (2/98) for optical
+   inputs to match the CROMA contract. That is a **deliberate, documented choice**, not an upstream
+   fact.
+2. The local subset is **100 % single-label**, against the official 1–11 multi-label scheme, so
+   metrics computed on it are **not comparable** to published multi-label numbers.
+
+---
+
+## 9. Where the evidence lives
+
+| Topic | Evidence |
+|---|---|
+| Findings F4-1…F5-3, C-1…C-9, P7-1 | `docs/PHASE*.md`, `docs/ARCHITECTURE_*.md`, `docs/CROMA_NORMALISATION_UPSTREAM_EVIDENCE.md` |
+| The 448-vs-224 paired test | `docs/PHASE7_RESOLUTION_DECISION.md` |
+| The VLM rejection | `docs/PHASE6_RUN1_REJECTION_DIAGNOSIS.md`, `artifacts/vlm/phase6_closure.json` |
+| Router defect + 3 live passes | `.workbuddy-ai/scratch/live_validation/` (`run_output.txt`, `run_final2.txt`, `run_final3.txt`) |
+| The undeployed B-07 patch | session scratch `fix-b07-forward-unavailable.patch` |
+| Live validation harness | `.workbuddy-ai/scratch/run_all_postfix2.harness`, `recompute_verdicts.py` |