Horos v2: macro-F1 0.613 (was 0.512), artifact-concrete realizations + restored concept negatives

README.md

@@ -21,6 +21,11 @@ datasets:
 [Topos](https://github.com/dialoguesai/topos): free text in → the data scopes it touches,
 an explicit `none`, or a hand-off to an LLM when it isn't sure.
 
+**v2** (2026-08-16) — macro-F1 **0.613**, up from v1's 0.512. Trained against a gap v1's
+own failures exposed: it handled *band* phrasings ("how has my sleep been") and fell
+silent on *artifact* ones ("what's my resting heart rate trend"). Still not promoted —
+see Limitations.
+
 ![How Horos routes a question](routing.png)
 
 *ὅρος — the stones Athenians planted at property lines: "whose land begins here." Same
@@ -50,15 +55,18 @@ Benchmark: `role_classify_8` — 1,071 cases, template-disjoint from training, m
 gold, 24% adversarial negatives. It is built to be hard: don't anchor against CLINC-style
 intent scores. The right column is the same benchmark's LLM baseline.
 
-| metric | Horos | mistral:7b (4.4 GB) |
-|---|---|---|
-| macro-F1 | **0.512** | 0.495 |
-| exact set match | **0.496** | 0.243 |
-| negatives abstained | **0.984** | 0.126 |
-| single / multi-gold recall | 0.388 / 0.425 | — |
-| dead rate | 0.171 | — |
-| disjoint rate | 0.217 | — |
-| per-scope recall ≥ 0.60 | 2 / 14 | 9 / 14 |
+| metric | v2 | v1 | mistral:7b (4.4 GB) |
+|---|---|---|---|
+| macro-F1 | **0.613** | 0.512 | 0.495 |
+| exact set match | **0.558** | 0.496 | 0.243 |
+| negatives abstained | **0.941** | 0.984 | 0.126 |
+| single / multi-gold recall | 0.502 / 0.596 | 0.388 / 0.425 | — |
+| dead rate | 0.149 | 0.171 | — |
+| disjoint rate | 0.206 | 0.217 | — |
+| per-scope recall ≥ 0.60 | 6 / 14 | 2 / 14 | 9 / 14 |
+
+On a 53-case hand-annotated probe of natural phrasings — which neither version trained
+on — v2 routes **77%** correctly against v1's 58%.
 
 ### With its LLM escalation (composed per-case)
 
@@ -75,11 +83,18 @@ traffic degrades with weaker hardware.
 
 ## Limitations
 
-* **Confident-none swallowing.** 27% of benchmark positives get a confident `none` —
-  silently, no escalation. The dominant defect; mechanism is template familiarity.
-  Treat a `none` on a plausibly-personal question with suspicion.
-* **12 of 14 scopes are under the 0.60 recall floor.** This artifact has not cleared its
-  promotion gate; it fronts an LLM in shadow/advisory postures only.
+* **Confident-none swallowing, now concentrated rather than general.** Overall dead rate
+  is 0.149, but on *artifact-concrete* phrasings ("what's my bank balance", "what's in
+  my review queue") it is **18%** against 5% on abstract ones. v2 closed most of v1's
+  26% gap here and gave part of it back for the safety property below — one threshold
+  currently trades them against each other, which is the next fix. Treat a `none` on a
+  plausibly-personal question with suspicion.
+* **Disjoint rate 0.206 — the blocking defect, and unsolved.** One in five acted-on
+  turns names a scope set sharing nothing with the truth. It is **threshold-invariant**
+  (0.256–0.270 across τ 0.4–0.9 on a sibling run), so it lives in the weights, not the
+  operating point. Three training rounds have not moved it.
+* **8 of 14 scopes are under the 0.60 recall floor** (v1: 12). This artifact has not
+  cleared its promotion gate; it fronts an LLM in shadow/advisory postures only.
 * All numbers are synthetic-benchmark. Real-traffic behaviour is being measured in
   shadow mode.
 * English only. **No user data, ever** — the loader refuses artifacts whose manifest
@@ -89,9 +104,16 @@ traffic degrades with weaker hardware.
 
 | source | licence | rows |
 |---|---|---|
-| schema-grounded synthetic + compounds + boundary negatives (Topos) | internal | 5,164 |
-| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | 3,500 |
-| [CLINC150 OOS](https://github.com/clinc/oos-eval) | CC BY 3.0 | 800 |
+| schema-grounded synthetic + compounds + boundary negatives (Topos) | internal | 7,369 |
+| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | 5,200 |
+| [CLINC150 OOS](https://github.com/clinc/oos-eval) | CC BY 3.0 | 1,200 |
+
+v2 added ~95 artifact-concrete realizations **and** restored the concept-negative ratio
+(0.054 → 0.070). The second half is the interesting one: without a "what *is* a
+curriculum vitae?" negative beside "what's on *my* CV?", an earlier v2 candidate learned
+the artifact noun as sufficient evidence of ownership and fired on 43 near-miss
+definition questions, dropping negatives-abstained to 0.739. Teaching a positive without
+its matching negative was the whole regression.
 
 This card is the CC BY attribution notice. No share-alike or non-commercial data.
 

head.json

@@ -29,46 +29,46 @@
         "source": "schema-grounded (G3)",
         "licence": "internal",
         "url": "-",
-        "rows": 1927,
-        "note": "the only source for the nine uncovered scopes; `train` split only (1927 of 4447) \u2014 the rest belongs to the benchmark"
+        "rows": 2767,
+        "note": "the only source for the nine uncovered scopes; `train` split only (2767 of 5987) \u2014 the rest belongs to the benchmark"
       },
       {
         "source": "compound positives (B1)",
         "licence": "internal",
         "url": "-",
-        "rows": 1772,
+        "rows": 2545,
         "note": "multi-scope conjunctions of train-split parents at 0.92x positives; the corpus was 100% single-label and trained AGAINST co-activation (multi-gold recall 0.165 vs 0.356 single)"
       },
       {
         "source": "negative twins (G4)",
         "licence": "internal",
         "url": "-",
-        "rows": 1156,
-        "note": "third-party + generalize at 0.60x positives (2461 available); teaches 'whose data', which M1 found rung 1 cannot represent"
+        "rows": 1660,
+        "note": "third-party + generalize at 0.60x positives (3544 available); teaches 'whose data', which M1 found rung 1 cannot represent"
       },
       {
         "source": "concept negatives (G4b)",
         "licence": "internal",
         "url": "-",
-        "rows": 309,
+        "rows": 397,
         "note": "mechanism + definition frames over the positives' own concept vocabulary; replaces the mined QA band, which yielded 102 usable rows from 119,700 questions and reached only 8 of 14 scopes"
       },
       {
         "source": "AmazonScience/massive",
         "licence": "CC BY-4.0",
         "url": "https://amazon-massive-nlu-dataset.s3.amazonaws.com/amazon-massive-dataset-1.1.tar.gz",
-        "rows": 3500,
+        "rows": 5200,
         "note": "en-US only; positives cue-filtered (see POSITIVE_CUES)"
       },
       {
         "source": "clinc/oos-eval",
         "licence": "CC BY-3.0",
         "url": "https://raw.githubusercontent.com/clinc/oos-eval/master/data/data_full.json",
-        "rows": 800,
+        "rows": 1200,
         "note": "all `none`; the OOS split is the near-miss anchor"
       }
     ],
-    "total_rows": 9464,
+    "total_rows": 13769,
     "attribution_required": [
       "AmazonScience/massive",
       "clinc/oos-eval"
@@ -76,29 +76,25 @@
   },
   "metrics": {
     "n": 1071,
-    "macro_f1": 0.5116596438030812,
-    "exact": 0.4957983193277311,
-    "negatives_abstained": 0.9841897233201581,
+    "macro_f1": 0.6127984463644067,
+    "exact": 0.5583566760037348,
+    "negatives_abstained": 0.9407114624505929,
     "scopes_below_floor": [
-      "activity:read",
       "ai_conversations:read",
-      "attention:read",
       "availability:read",
       "complexity:read",
       "health:read",
-      "messages:read",
       "places:read",
       "relationship_context:read",
       "resources:read",
-      "schedule:read",
       "work_context:read"
     ],
-    "recall_single_gold": 0.38816738816738816,
-    "recall_multi_gold": 0.4245614035087719,
-    "multi_gap": -0.036394015341383756,
-    "dead_rate": 0.17114914425427874,
-    "disjoint_rate": 0.21710526315789475,
+    "recall_single_gold": 0.5021645021645021,
+    "recall_multi_gold": 0.5964912280701754,
+    "multi_gap": -0.09432672590567326,
+    "dead_rate": 0.1491442542787286,
+    "disjoint_rate": 0.2063758389261745,
     "threshold": 0.7
   },
-  "trained_at": "2026-08-15"
+  "trained_at": "2026-08-16"
 }

model/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:91a54633cd655bc8ca3cd71e9a90088136576d0fb2589d333167f8ce54f372a1
+oid sha256:470b940f751055aae58c8f08ec045e0ae6bc69567ae09962a7adce1fe4ad3d24
 size 267872556