card: block 4 was v1's composition — recompute for v2, add CIs and ratchets

README.md

@@ -42,6 +42,13 @@ The `scope` role in a Topos model pack. Load through `topos.query.scope_head.loa
 against the licence policy before touching a weight. Not intended for any task besides
 Topos scope routing, and not intended to run without the escalation path.
 
+**Pin a revision.** The thresholds in `head.json` are selected per-artifact and the label
+set can change; `main` is not a stable contract. Topos pins the SHA it was gated against:
+
+```python
+snapshot_download("Dialogues/horos", revision="1873a1b667dc2d1bd630835f1f4f47025f3864d9")
+```
+
 ```
 labels ≥ tau_high, none low, nothing in band → ACT on the scope set (1..N scopes)
 high none, nothing else                      → abstain: no personal data
@@ -52,20 +59,27 @@ nothing anywhere, none included              → escalate to the pack's LLM (ign
 ## Report card
 
 Every release publishes all four blocks below, in full, **including the numbers that got
-worse** — this is a standing contract, not a summary of a good run. Each block is here
-because a metric nobody was required to report hid a real regression:
-
-* `disjoint` sat at 0.206–0.265 across three training rounds while macro-F1 climbed 20%.
-  An average would have called that pure progress.
-* Artifact-concrete swallowing was 26% while the overall dead rate looked healthy at
-  0.171. The split found the defect; the total hid it.
-* `wrong` on unseen phrasings rose 11% → 17% in the same round routing improved 14
-  points. Only block 3 can see that.
-* `swallowed` and `escalated` have opposite fixes. Collapsing them into "not answered"
-  hides which one you have.
-
-Reproduce every block with one command:
-`topos-eval/scripts/scope_head_report_card.py --head A --label v1 --head B --label v2`.
+worse** — this is a standing contract, not a summary of a good run. Every row is versioned
+and every cross-arm claim carries an interval. Each block earns its slot by having caught
+something the others hid:
+
+* **1 — the gate.** `disjoint` sat at 0.206–0.265 across three training rounds while
+  macro-F1 climbed 20%. An average would have called that pure progress.
+* **2 — the split.** Artifact-concrete swallowing was 26% while the overall dead rate
+  looked healthy at 0.171. The total hid a defect the split names.
+* **3 — unseen data.** `wrong` rose 11% → 17% in the same round routing improved 14
+  points. Nothing else here can see a model converting silence into error.
+* **4 — composed.** Horos never runs alone; a head-only number is not a product number.
+  It is also where an unqualified 0.500-vs-0.495 claim went out on this card and turned
+  out to be inside the noise (see block 4).
+
+`swallowed` and `escalated` stay separate throughout: they have opposite fixes, and
+collapsing them into "not answered" hides which one you have.
+
+Blocks 1–3 regenerate from one command in `topos-eval`
+(`scripts/scope_head_report_card.py --head A --label v1 --head B --label v2`). That
+repo is private, so the numbers here are auditable by us and take-it-or-leave-it for
+you — the benchmark itself is what we'd have to open to change that.
 
 ### 1. `role_classify_8` — the promotion gate
 
@@ -103,36 +117,51 @@ is the signal worth acting on.
 never draws from. Provably unseen by both versions and absent from the benchmark. This
 block separates *learning a register* from *memorising the rows we wrote*.
 
-| metric | v2 | v1 |
-|---|---|---|
-| routed | **0.565** | 0.421 |
-| swallowed | **0.172** | 0.214 |
-| **wrong scope** | 0.169 | 0.113 |
-| escalated | 0.094 | 0.252 |
+| metric | v2 | v1 | ratchet |
+|---|---|---|---|
+| routed | **0.565** | 0.421 | ≥ incumbent ✅ |
+| swallowed | **0.172** | 0.214 | ≤ incumbent ✅ |
+| **wrong scope** | 0.169 | 0.113 | ≤ incumbent ❌ |
+| escalated | 0.094 | 0.252 | — |
 
 **Read the `wrong` row, not just the first one.** v2 routes 14 points more of this
 traffic and is confidently wrong 6 points more often. It got better at recognising that
 a question concerns personal data, and no better at discriminating *which* data — the
 same finding as the flat `disjoint` rate. Those are separate abilities; one improved.
 
-### 4. With its LLM escalation (composed per-case)
+### 4. Composed with its LLM escalation
+
+Horos is not deployed alone, so this is the number that describes the product. Each case
+is routed by the ladder above: Horos answers when it acts or abstains, the LLM sees only
+what it escalates.
 
 | arm | macro-F1 | exact | neg-abstain | wrong-scope | LLM sees |
 |---|---|---|---|---|---|
-| mistral:7b alone | 0.495 | 0.243 | 0.126 | 0.237 | 100% |
-| **Horos + mistral:7b** | **0.550** | **0.524** | **0.972** | **0.143** | **16.4%** |
+| mistral:7b (4.4 GB) alone | 0.495 | 0.243 | 0.126 | 0.237 | 100% |
+| Horos **v1** + mistral:7b | 0.550 | 0.524 | 0.972 | 0.143 | 16.4% |
+| **Horos v2 + mistral:7b** | **0.620** | **0.578** | 0.909 | 0.163 | **15.3%** |
 | llama3.2 (2 GB) alone | 0.376 | 0.261 | 0.510 | 0.204 | 100% |
-| **Horos + llama3.2** | **0.500** | **0.508** | **0.976** | 0.146 | 16.4% |
+| Horos **v1** + llama3.2 | 0.500 | 0.508 | 0.976 | 0.146 | 16.4% |
+| **Horos v2 + llama3.2** | **0.583** | **0.565** | 0.917 | 0.146 | **15.3%** |
 
-A 2 GB machine running the hybrid (0.500) beats a 4.4 GB machine running LLM-only
-(0.495): Horos is identical on every hardware tier, so only the escalated ~1/6th of
+**A 2 GB machine running the hybrid beats a 4.4 GB machine running LLM-only** — 0.583 vs
+0.495, **+0.088, 95% CI [+0.052, +0.123]** (paired bootstrap, 2,000 resamples of the same
+1,071 cases). Horos is identical on every hardware tier, so only the escalated ~1/6th of
 traffic degrades with weaker hardware.
 
+That claim did not survive this test at v1, where the same comparison was +0.005, CI
+[−0.036, +0.042] — indistinguishable from zero. It was stated on this card as fact for
+one release. Cross-arm claims here now carry an interval or they don't get made.
+
+Note the direction of the v1 → v2 trade in this table: composed accuracy up ~7 points,
+composed negatives-abstained down ~6. The escalation path was covering for the head's
+false-positives, and v2 hands it less to cover.
+
 ## Limitations
 
 * **Confident-none swallowing, now concentrated rather than general.** Overall dead rate
   is 0.149, but on *artifact-concrete* phrasings ("what's my bank balance", "what's in
-  my review queue") it is **18%** against 5% on abstract ones. v2 closed most of v1's
+  my review queue") it is **17.6%** against 5.3% on abstract ones. v2 closed most of v1's
   26% gap here and gave part of it back for the safety property below — one threshold
   currently trades them against each other, which is the next fix. Treat a `none` on a
   plausibly-personal question with suspicion.
@@ -140,6 +169,12 @@ traffic degrades with weaker hardware.
   turns names a scope set sharing nothing with the truth. It is **threshold-invariant**
   (0.256–0.270 across τ 0.4–0.9 on a sibling run), so it lives in the weights, not the
   operating point. Three training rounds have not moved it.
+* **v2 gained recognition, not discrimination.** It routes far more personal-data
+  questions than v1 and is confidently wrong on a larger share of them (block 3: `wrong`
+  0.113 → 0.169, against a flat `disjoint`). Knowing a question is about your data and
+  knowing *which* of your data are separate abilities; this round only advanced the
+  first. The escalation path absorbs less of that than it used to — composed
+  negatives-abstained fell 0.972 → 0.909.
 * **8 of 14 scopes are under the 0.60 recall floor** (v1: 12). This artifact has not
   cleared its promotion gate; it fronts an LLM in shadow/advisory postures only.
 * All numbers are synthetic-benchmark. Real-traffic behaviour is being measured in