card: standing report-card contract — four blocks published every release, regressions included

README.md

@@ -49,26 +49,73 @@ any label in [tau_low, tau_high)             → escalate to the pack's LLM (amb
 nothing anywhere, none included              → escalate to the pack's LLM (ignorance)
 ```
 
-## Numbers
+## Report card
+
+Every release publishes all four blocks below, in full, **including the numbers that got
+worse** — this is a standing contract, not a summary of a good run. Each block is here
+because a metric nobody was required to report hid a real regression:
+
+* `disjoint` sat at 0.206–0.265 across three training rounds while macro-F1 climbed 20%.
+  An average would have called that pure progress.
+* Artifact-concrete swallowing was 26% while the overall dead rate looked healthy at
+  0.171. The split found the defect; the total hid it.
+* `wrong` on unseen phrasings rose 11% → 17% in the same round routing improved 14
+  points. Only block 3 can see that.
+* `swallowed` and `escalated` have opposite fixes. Collapsing them into "not answered"
+  hides which one you have.
+
+Reproduce every block with one command:
+`topos-eval/scripts/scope_head_report_card.py --head A --label v1 --head B --label v2`.
+
+### 1. `role_classify_8` — the promotion gate
+
+1,071 cases, template-disjoint from training, multi-label gold, 24% adversarial
+negatives. Built to be hard: don't anchor against CLINC-style intent scores. The right
+column is the same benchmark's LLM baseline.
+
+| metric | v2 | v1 | mistral:7b (4.4 GB) | gate |
+|---|---|---|---|---|
+| macro-F1 | **0.613** | 0.512 | 0.495 | ≥ incumbent ✅ |
+| exact set match | **0.558** | 0.496 | 0.243 | — |
+| negatives abstained | 0.941 | 0.984 | 0.126 | ≥0.85 ✅ |
+| single / multi-gold recall | 0.502 / 0.596 | 0.388 / 0.425 | — | gap ≤0.05 ✅ |
+| dead rate | **0.149** | 0.171 | — | <0.20 ✅ |
+| **disjoint rate** | 0.206 | 0.217 | — | ≤0.03 ❌ |
+| per-scope recall ≥ 0.60 | 6 / 14 | 2 / 14 | 9 / 14 | 14/14 ❌ |
+
+### 2. Real language — a gap finder, not a score
+
+53 hand-annotated natural phrasings neither version trained on. The labels are one
+annotator's judgement, so the totals measure agreement-with-the-annotator; the **split**
+is the signal worth acting on.
+
+| metric | v2 | v1 | gate |
+|---|---|---|---|
+| correct | **0.774** | 0.585 | — |
+| swallowed | **0.132** | 0.189 | — |
+| escalated | 0.094 | 0.189 | — |
+| **swallowed — artifact-concrete** | 0.176 | 0.265 | ≤0.10 ❌ |
+| swallowed — abstract / band | 0.053 | 0.053 | — |
 
-Benchmark: `role_classify_8` — 1,071 cases, template-disjoint from training, multi-label
-gold, 24% adversarial negatives. It is built to be hard: don't anchor against CLINC-style
-intent scores. The right column is the same benchmark's LLM baseline.
+### 3. Unseen phrasings — did the authoring generalise?
 
-| metric | v2 | v1 | mistral:7b (4.4 GB) |
-|---|---|---|---|
-| macro-F1 | **0.613** | 0.512 | 0.495 |
-| exact set match | **0.558** | 0.496 | 0.243 |
-| negatives abstained | **0.941** | 0.984 | 0.126 |
-| single / multi-gold recall | 0.502 / 0.596 | 0.388 / 0.425 | — |
-| dead rate | 0.149 | 0.171 | — |
-| disjoint rate | 0.206 | 0.217 | — |
-| per-scope recall ≥ 0.60 | 6 / 14 | 2 / 14 | 9 / 14 |
-
-On a 53-case hand-annotated probe of natural phrasings — which neither version trained
-on — v2 routes **77%** correctly against v1's 58%.
-
-### With its LLM escalation (composed per-case)
+2,745 rows from the schema-grounded generator's heldout half, which the training corpus
+never draws from. Provably unseen by both versions and absent from the benchmark. This
+block separates *learning a register* from *memorising the rows we wrote*.
+
+| metric | v2 | v1 |
+|---|---|---|
+| routed | **0.565** | 0.421 |
+| swallowed | **0.172** | 0.214 |
+| **wrong scope** | 0.169 | 0.113 |
+| escalated | 0.094 | 0.252 |
+
+**Read the `wrong` row, not just the first one.** v2 routes 14 points more of this
+traffic and is confidently wrong 6 points more often. It got better at recognising that
+a question concerns personal data, and no better at discriminating *which* data — the
+same finding as the flat `disjoint` rate. Those are separate abilities; one improved.
+
+### 4. With its LLM escalation (composed per-case)
 
 | arm | macro-F1 | exact | neg-abstain | wrong-scope | LLM sees |
 |---|---|---|---|---|---|