scope head b4: macro-F1 0.512 on classify-8, 15-label (14 scopes + none), trained 2026-08-15

README.md

@@ -0,0 +1,125 @@
+---
+license: apache-2.0
+language:
+- en
+base_model: distilbert-base-uncased
+pipeline_tag: text-classification
+tags:
+- topos
+- scope-routing
+- multi-label
+- on-device
+- privacy
+datasets:
+- AmazonScience/massive
+- clinc/oos-eval
+---
+
+# Topos Scope Head
+
+An on-device, multi-label scope router for [Topos](https://github.com/dialoguesai/topos).
+Given a free-text question, it predicts which UMA personal-data **scopes** the question
+touches (e.g. `health:read`, `schedule:read`), predicts an explicit **`none`** class for
+questions that touch no personal data, or signals **uncertainty** so the caller can
+escalate to an LLM. It runs *before* the permission gate — the raw question is the one
+input the pipeline sees pre-grant, which is why this model exists: so that step never
+has to leave the device.
+
+**This model is a router, not an authorization mechanism.** Every scope it emits is
+still permission-checked downstream. A node without this model escalates to an LLM more
+often; it does not become less safe.
+
+## Intended use
+
+Deployed as the `scope` role in a Topos model pack, behind a four-branch ladder:
+
+```
+labels ≥ tau_high, none low, nothing in band → ACT on the scope set (1..N scopes)
+high none, nothing else                      → confident abstain (no personal data)
+any label in [tau_low, tau_high)             → ESCALATE to the pack's LLM (ambiguity)
+nothing anywhere, none included              → ESCALATE to the pack's LLM (ignorance)
+```
+
+It is **not** intended to be used without the escalation path, and not intended for any
+classification task other than Topos scope routing (its 14 scope labels + `none` are the
+Topos scope registry, versioned; the loader refuses the artifact if the live registry
+has drifted).
+
+## Numbers, with the method
+
+All measured on `role_classify_8` (1,071 cases, 818 positive / 253 negative), template-
+disjoint from training, threshold 0.70 selected on a pre-training grouped holdout —
+never on this benchmark.
+
+**Do not anchor these numbers against standard intent benchmarks.** This benchmark is
+deliberately adversarial: every positive uses phrasing templates the model never saw,
+gold is multi-label, and 24% of cases are near-miss negatives (third-party probes,
+mechanism/definition traps) built to punish keyword matching. For scale on the *same*
+benchmark: **mistral:7b, a 4.4 GB LLM, scores 0.495 macro-F1** — this 265 MB model beats
+it while running every turn on-device.
+
+| metric | value | mistral:7b, same benchmark |
+|---|---|---|
+| macro-F1 | **0.512** | 0.495 |
+| exact set match | **0.496** | 0.243 |
+| negatives abstained | **0.984** | 0.126 |
+| single-gold vs multi-gold recall | 0.388 / 0.425 (gap −0.036) | — |
+| dead rate (no opinion incl. `none`) | 0.171 | — |
+| disjoint rate (acted, set shares nothing with gold) | 0.217 | — |
+| per-scope recall ≥ 0.60 | 2 / 14 | 9 / 14 |
+
+### As the hybrid it is designed to front (head + LLM escalation)
+
+Composed per-case: the head's verdict stands on act/abstain; escalated turns take the
+LLM's actual answer on those exact cases.
+
+| arm | macro-F1 | exact | neg-abstain | wrong-scope/turns | LLM sees |
+|---|---|---|---|---|---|
+| LLM-only mistral:7b (4.4 GB) | 0.495 | 0.243 | 0.126 | 0.237 | 100% |
+| **hybrid: this model + mistral:7b** | **0.550** | **0.524** | **0.972** | **0.143** | **16.4%** |
+| LLM-only llama3.2 (2 GB) | 0.376 | 0.261 | 0.510 | 0.204 | 100% |
+| **hybrid: this model + llama3.2** | **0.500** | **0.508** | **0.976** | 0.146 | 16.4% |
+
+The result that matters for low-RAM machines: **a 2 GB-LLM machine running the hybrid
+(0.500) outperforms a 4.4 GB-LLM machine running LLM-only (0.495)** — the head is 265 MB
+and identical on every hardware tier, so only ~1/6th of traffic degrades with weaker
+hardware.
+
+## Known limitations — read before relying on it
+
+* **Confident-none swallowing.** 222 of 818 benchmark positives (27%) are answered with
+  a confident `none` — silently, with no escalation. This is the model's dominant defect.
+  Measured mechanism: template familiarity (heldout phrasings fall to `none`), not class
+  balance. Callers should treat a `none` on a plausibly-personal question with suspicion.
+* **Per-scope recall is far below production floors** (12 of 14 scopes under 0.60).
+  This artifact has **not** cleared its promotion gate and fronts an LLM in shadow /
+  advisory postures only.
+* All numbers above are **synthetic-benchmark** numbers. Real-traffic behaviour is being
+  measured in shadow mode; median confidence on real language is substantially lower
+  than on generated text.
+* English only. Trained entirely on synthetic templates + the public corpora below —
+  **no user data, ever** (the Topos loader refuses artifacts whose manifest violates
+  this).
+
+## Training data & attribution
+
+| source | licence | rows | role |
+|---|---|---|---|
+| schema-grounded synthetic (Topos) | internal/synthetic | 1,927 | positives |
+| compound conjunctions of the above | internal/synthetic | 1,772 | multi-scope positives |
+| negative twins + concept negatives | internal/synthetic | 1,465 | boundary negatives |
+| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | 3,500 | real-register `none` + cue-mapped positives |
+| [CLINC150 OOS](https://github.com/clinc/oos-eval) | CC BY 3.0 | 800 | out-of-scope `none` |
+
+This model card is the attribution notice CC BY requires. No share-alike or
+non-commercial data was used; the Topos loader enforces that class of licence refusal
+at load time.
+
+## Architecture
+
+DistilBERT-base-uncased, 15-label multi-label head (14 scopes + `none`), sigmoid
+outputs, `BCEWithLogitsLoss` with per-label `pos_weight`, max_length 64. Artifact format
+`topos-scope-head-1`: `head.json` (labels, thresholds, corpus manifest, metrics) +
+`model/` (HF-standard). Load through `topos.query.scope_head.load_head`, which validates
+labels against the live scope registry and the manifest against the licence policy
+before any weight is touched.

head.json

@@ -0,0 +1,104 @@
+{
+  "format": "topos-scope-head-1",
+  "kind": "encoder",
+  "labels": [
+    "activity:read",
+    "ai_conversations:read",
+    "attention:read",
+    "availability:read",
+    "complexity:read",
+    "contacts:resolve",
+    "health:read",
+    "messages:read",
+    "places:read",
+    "public_bio:read",
+    "relationship_context:read",
+    "resources:read",
+    "schedule:read",
+    "work_context:read",
+    "none"
+  ],
+  "base_model": "distilbert-base-uncased",
+  "embedding_model": "",
+  "tau_high": 0.7,
+  "tau_low": 0.5499999999999999,
+  "max_length": 64,
+  "corpus_manifest": {
+    "corpora": [
+      {
+        "source": "schema-grounded (G3)",
+        "licence": "internal",
+        "url": "-",
+        "rows": 1927,
+        "note": "the only source for the nine uncovered scopes; `train` split only (1927 of 4447) \u2014 the rest belongs to the benchmark"
+      },
+      {
+        "source": "compound positives (B1)",
+        "licence": "internal",
+        "url": "-",
+        "rows": 1772,
+        "note": "multi-scope conjunctions of train-split parents at 0.92x positives; the corpus was 100% single-label and trained AGAINST co-activation (multi-gold recall 0.165 vs 0.356 single)"
+      },
+      {
+        "source": "negative twins (G4)",
+        "licence": "internal",
+        "url": "-",
+        "rows": 1156,
+        "note": "third-party + generalize at 0.60x positives (2461 available); teaches 'whose data', which M1 found rung 1 cannot represent"
+      },
+      {
+        "source": "concept negatives (G4b)",
+        "licence": "internal",
+        "url": "-",
+        "rows": 309,
+        "note": "mechanism + definition frames over the positives' own concept vocabulary; replaces the mined QA band, which yielded 102 usable rows from 119,700 questions and reached only 8 of 14 scopes"
+      },
+      {
+        "source": "AmazonScience/massive",
+        "licence": "CC BY-4.0",
+        "url": "https://amazon-massive-nlu-dataset.s3.amazonaws.com/amazon-massive-dataset-1.1.tar.gz",
+        "rows": 3500,
+        "note": "en-US only; positives cue-filtered (see POSITIVE_CUES)"
+      },
+      {
+        "source": "clinc/oos-eval",
+        "licence": "CC BY-3.0",
+        "url": "https://raw.githubusercontent.com/clinc/oos-eval/master/data/data_full.json",
+        "rows": 800,
+        "note": "all `none`; the OOS split is the near-miss anchor"
+      }
+    ],
+    "total_rows": 9464,
+    "attribution_required": [
+      "AmazonScience/massive",
+      "clinc/oos-eval"
+    ]
+  },
+  "metrics": {
+    "n": 1071,
+    "macro_f1": 0.5116596438030812,
+    "exact": 0.4957983193277311,
+    "negatives_abstained": 0.9841897233201581,
+    "scopes_below_floor": [
+      "activity:read",
+      "ai_conversations:read",
+      "attention:read",
+      "availability:read",
+      "complexity:read",
+      "health:read",
+      "messages:read",
+      "places:read",
+      "relationship_context:read",
+      "resources:read",
+      "schedule:read",
+      "work_context:read"
+    ],
+    "recall_single_gold": 0.38816738816738816,
+    "recall_multi_gold": 0.4245614035087719,
+    "multi_gap": -0.036394015341383756,
+    "dead_rate": 0.17114914425427874,
+    "disjoint_rate": 0.21710526315789475,
+    "threshold": 0.7
+  },
+  "trained_at": "2026-08-15"
+}

model/config.json

@@ -0,0 +1,61 @@
+{
+  "activation": "gelu",
+  "architectures": [
+    "DistilBertForSequenceClassification"
+  ],
+  "attention_dropout": 0.1,
+  "bos_token_id": null,
+  "dim": 768,
+  "dropout": 0.1,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_dim": 3072,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2",
+    "3": "LABEL_3",
+    "4": "LABEL_4",
+    "5": "LABEL_5",
+    "6": "LABEL_6",
+    "7": "LABEL_7",
+    "8": "LABEL_8",
+    "9": "LABEL_9",
+    "10": "LABEL_10",
+    "11": "LABEL_11",
+    "12": "LABEL_12",
+    "13": "LABEL_13",
+    "14": "LABEL_14"
+  },
+  "initializer_range": 0.02,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_10": 10,
+    "LABEL_11": 11,
+    "LABEL_12": 12,
+    "LABEL_13": 13,
+    "LABEL_14": 14,
+    "LABEL_2": 2,
+    "LABEL_3": 3,
+    "LABEL_4": 4,
+    "LABEL_5": 5,
+    "LABEL_6": 6,
+    "LABEL_7": 7,
+    "LABEL_8": 8,
+    "LABEL_9": 9
+  },
+  "max_position_embeddings": 512,
+  "model_type": "distilbert",
+  "n_heads": 12,
+  "n_layers": 6,
+  "pad_token_id": 0,
+  "problem_type": "multi_label_classification",
+  "qa_dropout": 0.1,
+  "seq_classif_dropout": 0.2,
+  "sinusoidal_pos_embds": false,
+  "tie_weights_": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.10.2",
+  "vocab_size": 30522
+}

model/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91a54633cd655bc8ca3cd71e9a90088136576d0fb2589d333167f8ce54f372a1
+size 267872556

model/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}