README.md

---
license: apache-2.0
language:
- en
base_model: distilbert-base-uncased
pipeline_tag: text-classification
tags:
- topos
- scope-routing
- multi-label
- on-device
- privacy
datasets:
- AmazonScience/massive
- clinc/oos-eval
---

# Horos

**The boundary stone for personal data.** A 268 MB on-device router for
[Topos](https://github.com/dialoguesai/topos): free text in → the data scopes it touches,
an explicit `none`, or a hand-off to an LLM when it isn't sure.

**v2** (2026-08-16) — macro-F1 **0.613**, up from v1's 0.512. Trained against a gap v1's
own failures exposed: it handled *band* phrasings ("how has my sleep been") and fell
silent on *artifact* ones ("what's my resting heart rate trend"). Still not promoted —
see Limitations.

![How Horos routes a question](routing.png)

*ὅρος — the stones Athenians planted at property lines: "whose land begins here." Same
job, for your data. Horos runs before the permission gate, so the raw question never
leaves the device.*

**Router, not authorizer.** Every scope it emits is still permission-checked downstream.
A node without Horos escalates more often; it does not become less safe.

## Use

The `scope` role in a Topos model pack. Load through `topos.query.scope_head.load_head`
— it validates the label set against the live scope registry and the training manifest
against the licence policy before touching a weight. Not intended for any task besides
Topos scope routing, and not intended to run without the escalation path.

**Pin a revision.** The thresholds in `head.json` are selected per-artifact and the label
set can change; `main` is not a stable contract. Topos pins the SHA it was gated against:

```python
snapshot_download("Dialogues/horos", revision="1873a1b667dc2d1bd630835f1f4f47025f3864d9")
```

**The ladder.** Four branches, because "no scopes" is two different states — the model
deciding there is no personal data here, and the model having no idea:

```
labels ≥ tau_high, none low, nothing in band → ACT on the scope set (1..N scopes)
high none, nothing else                      → abstain: no personal data
any label in [tau_low, tau_high)             → escalate to the pack's LLM (ambiguity)
nothing anywhere, none included              → escalate to the pack's LLM (ignorance)
```

## Report card

Every release publishes all five blocks below, in full, **including the numbers that got
worse** — this is a standing contract, not a summary of a good run. Every row is versioned
and every cross-arm claim carries an interval. Each block earns its slot by having caught
something the others hid:

* **1 — the gate.** `disjoint` sat at 0.206–0.265 across three training rounds while
  macro-F1 climbed 20%. An average would have called that pure progress.
* **2 — the split.** Artifact-concrete swallowing was 26% while the overall dead rate
  looked healthy at 0.171. The total hid a defect the split names.
* **3 — unseen data.** `wrong` rose 11% → 17% in the same round routing improved 14
  points. Nothing else here can see a model converting silence into error.
* **4 — composed.** Horos never runs alone; a head-only number is not a product number.
  It is also where an unqualified 0.500-vs-0.495 claim went out on this card and turned
  out to be inside the noise (see block 4).
* **5 — per-scope.** Every other block is an average or a count, and both let a gain on
  one scope pay for a regression on another without saying so. Only this view names the
  scope that got worse.

`swallowed` and `escalated` stay separate throughout: they have opposite fixes, and
collapsing them into "not answered" hides which one you have.

Blocks 1–3 and 5 regenerate from one command in `topos-eval`
(`scripts/scope_head_report_card.py --head A --label v1 --head B --label v2`). That
repo is private, so the numbers here are auditable by us and take-it-or-leave-it for
you — the benchmark itself is what we'd have to open to change that.

### 1. `role_classify_8` — the promotion gate

1,071 cases, template-disjoint from training, multi-label gold, 24% adversarial
negatives. Built to be hard: don't anchor against CLINC-style intent scores. The right
column is the same benchmark's LLM baseline.

| metric | v2 | v1 | mistral:7b (4.4 GB) | gate |
|---|---|---|---|---|
| macro-F1 | **0.613** | 0.512 | 0.495 | ≥ incumbent ✅ |
| exact set match | **0.558** | 0.496 | 0.243 | — |
| negatives abstained | 0.941 | 0.984 | 0.126 | ≥0.85 ✅ |
| single / multi-gold recall | 0.502 / 0.596 | 0.388 / 0.425 | — | gap ≤0.05 ✅ |
| dead rate | **0.149** | 0.171 | — | <0.20 ✅ |
| **disjoint rate** | 0.206 | 0.217 | — | ≤0.03 ❌ |
| per-scope recall ≥ 0.60 | 6 / 14 | 2 / 14 | 9 / 14 | 14/14 ❌ |

Which six, and which scope went backwards: block 5.

### 2. Real language — a gap finder, not a score

53 hand-annotated natural phrasings neither version trained on. The labels are one
annotator's judgement, so the totals measure agreement-with-the-annotator; the **split**
is the signal worth acting on.

| metric | v2 | v1 | gate |
|---|---|---|---|
| correct | **0.774** | 0.585 | — |
| swallowed | **0.132** | 0.189 | — |
| escalated | 0.094 | 0.189 | — |
| **swallowed — artifact-concrete** | 0.176 | 0.265 | ≤0.10 ❌ |
| swallowed — abstract / band | 0.053 | 0.053 | — |

### 3. Unseen phrasings — did the authoring generalise?

2,745 rows from the schema-grounded generator's heldout half, which the training corpus
never draws from. Provably unseen by both versions and absent from the benchmark. This
block separates *learning a register* from *memorising the rows we wrote*.

| metric | v2 | v1 | ratchet |
|---|---|---|---|
| routed | **0.565** | 0.421 | ≥ incumbent ✅ |
| swallowed | **0.172** | 0.214 | ≤ incumbent ✅ |
| **wrong scope** | 0.169 | 0.113 | ≤ incumbent ❌ |
| escalated | 0.094 | 0.252 | — |

**Read the `wrong` row, not just the first one.** v2 routes 14 points more of this
traffic and is confidently wrong 6 points more often. It got better at recognising that
a question concerns personal data, and no better at discriminating *which* data — the
same finding as the flat `disjoint` rate. Those are separate abilities; one improved.

### 4. Composed with its LLM escalation

Horos is not deployed alone, so this is the number that describes the product. Each case
is routed by the ladder above: Horos answers when it acts or abstains, the LLM sees only
what it escalates.

| arm | macro-F1 | exact | neg-abstain | wrong-scope | LLM sees |
|---|---|---|---|---|---|
| mistral:7b (4.4 GB) alone | 0.495 | 0.243 | 0.126 | 0.237 | 100% |
| Horos **v1** + mistral:7b | 0.550 | 0.524 | 0.972 | 0.143 | 16.4% |
| **Horos v2 + mistral:7b** | **0.620** | **0.578** | 0.909 | 0.163 | **15.3%** |
| llama3.2 (2 GB) alone | 0.376 | 0.261 | 0.510 | 0.204 | 100% |
| Horos **v1** + llama3.2 | 0.500 | 0.508 | 0.976 | 0.146 | 16.4% |
| **Horos v2 + llama3.2** | **0.583** | **0.565** | 0.917 | 0.146 | **15.3%** |

**A 2 GB machine running the hybrid beats a 4.4 GB machine running LLM-only** — 0.583 vs
0.495, **+0.088, 95% CI [+0.052, +0.123]** (paired bootstrap, 2,000 resamples of the same
1,071 cases). Horos is identical on every hardware tier, so only the escalated ~1/6th of
traffic degrades with weaker hardware.

That claim did not survive this test at v1, where the same comparison was +0.005, CI
[−0.036, +0.042] — indistinguishable from zero. It was stated on this card as fact for
one release. Cross-arm claims here now carry an interval or they don't get made.

Note the direction of the v1 → v2 trade in this table: composed accuracy up ~7 points,
composed negatives-abstained down ~6. The escalation path was covering for the head's
false-positives, and v2 hands it less to cover.

### 5. Per-scope — where it moved, and where it didn't

Blocks 1 and 3 report counts ("6 / 14 above the floor", "routed 0.565"). A count cannot
be audited: it says how many scopes cleared the bar and never which, so a large gain on
one scope silently pays for a regression on another. Both slices, sorted by movement.

**Unseen phrasings** — did the gains reach language nobody wrote down?

| scope | n | v1 | v2 | Δ |
|---|---|---|---|---|
| `places` | 217 | 30% | 68% | **+37** |
| `schedule` | 65 | 31% | 65% | **+34** |
| `work_context` | 420 | 22% | 51% | **+29** |
| `relationship_context` | 174 | 16% | 42% | **+26** |
| `public_bio` | 229 | 67% | 85% | +17 |
| `messages` | 253 | 49% | 66% | +17 |
| `attention` | 145 | 35% | 46% | +10 |
| `complexity` | 178 | 39% | 47% | +8 |
| `activity` | 308 | 61% | 68% | +7 |
| `resources` | 151 | 28% | 34% | +6 |
| `availability` | 116 | 21% | 25% | +4 |
| `contacts` | 119 | 76% | 73% | −3 |
| `ai_conversations` | 111 | 42% | 39% | −4 |
| `health` | 259 | 61% | 56% | **−5** |

**Gate benchmark** — this is what "6 / 14 above the floor" expands to.

| scope | n | v1 | v2 | Δ | ≥0.60 |
|---|---|---|---|---|---|
| `contacts` | 58 | 69% | 83% | +14 | ✅ |
| `public_bio` | 54 | 67% | 81% | +15 | ✅ |
| `attention` | 73 | 25% | 71% | **+47** | ✅ |
| `schedule` | 59 | 44% | 69% | +25 | ✅ |
| `activity` | 78 | 58% | 64% | +6 | ✅ |
| `messages` | 61 | 51% | 64% | +13 | ✅ |
| `health` | 116 | 47% | 54% | +8 | ❌ |
| `places` | 61 | 26% | 49% | +23 | ❌ |
| `availability` | 62 | 35% | 45% | +10 | ❌ |
| `resources` | 70 | 27% | 37% | +10 | ❌ |
| `complexity` | 82 | 35% | 37% | +1 | ❌ |
| `ai_conversations` | 54 | 43% | 35% | **−7** | ❌ |
| `relationship_context` | 77 | 17% | 34% | +17 | ❌ |
| `work_context` | 73 | 25% | 30% | +5 | ❌ |

Three things only this view shows:

* **`ai_conversations` is down on both slices** (−7, −4) — the one unambiguous regression,
  not a slice artifact. v2's training targeted artifact-concrete phrasings, and questions
  about your own past AI conversations are the scope least like an artifact.
* **`health` and `contacts` flip sign between slices.** `health` gains 8 on the benchmark
  and loses 5 on unseen phrasings; `contacts` gains 14 and loses 3. The two instruments
  measure genuinely different things, and a card publishing only one of them would report
  either as a clean win.
* **The four biggest unseen gains are exactly the four scopes v2's authoring targeted**
  (`places`, `schedule`, `work_context`, `relationship_context`, +26 to +37). That is the
  evidence the authoring generalised rather than being memorised — the gains land on
  phrasings of those scopes that nobody wrote down.

Floors are still unmet: on the gate slice, 8 of 14 scopes sit under the 0.60 recall bar.
`work_context` (30%), `complexity` (37%) and `resources` (37%) are the weakest, and
`relationship_context` at 34% remains the hardest scope in the taxonomy.

## Limitations

* **Confident-none swallowing, now concentrated rather than general.** Overall dead rate
  is 0.149, but on *artifact-concrete* phrasings ("what's my bank balance", "what's in
  my review queue") it is **17.6%** against 5.3% on abstract ones. v2 closed most of v1's
  26% gap here and gave part of it back for the safety property below — one threshold
  currently trades them against each other, which is the next fix. Treat a `none` on a
  plausibly-personal question with suspicion.
* **Disjoint rate 0.206 — the blocking defect, and unsolved.** One in five acted-on
  turns names a scope set sharing nothing with the truth. It is **threshold-invariant**
  (0.256–0.270 across τ 0.4–0.9 on a sibling run), so it lives in the weights, not the
  operating point. Three training rounds have not moved it.
* **v2 gained recognition, not discrimination.** It routes far more personal-data
  questions than v1 and is confidently wrong on a larger share of them (block 3: `wrong`
  0.113 → 0.169, against a flat `disjoint`). Knowing a question is about your data and
  knowing *which* of your data are separate abilities; this round only advanced the
  first. The escalation path absorbs less of that than it used to — composed
  negatives-abstained fell 0.972 → 0.909.
* **8 of 14 scopes are under the 0.60 recall floor** (v1: 12), weakest `work_context`
  30%, `relationship_context` 34%, `complexity` and `resources` 37%. This artifact has
  not cleared its promotion gate; it fronts an LLM in shadow/advisory postures only.
* **`ai_conversations` regressed on both slices** (−7 gate, −4 unseen) — the one scope
  v2 made unambiguously worse. Questions about your own past AI conversations are the
  scope least like the artifact-concrete register v2 was trained to fix.
* All numbers are synthetic-benchmark. Real-traffic behaviour is being measured in
  shadow mode.
* English only. **No user data, ever** — the loader refuses artifacts whose manifest
  says otherwise.

## Training data

| source | licence | rows |
|---|---|---|
| schema-grounded synthetic + compounds + boundary negatives (Topos) | internal | 7,369 |
| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | 5,200 |
| [CLINC150 OOS](https://github.com/clinc/oos-eval) | CC BY 3.0 | 1,200 |

v2 added ~95 artifact-concrete realizations **and** restored the concept-negative ratio
(0.054 → 0.070). The second half is the interesting one: without a "what *is* a
curriculum vitae?" negative beside "what's on *my* CV?", an earlier v2 candidate learned
the artifact noun as sufficient evidence of ownership and fired on 43 near-miss
definition questions, dropping negatives-abstained to 0.739. Teaching a positive without
its matching negative was the whole regression.

This card is the CC BY attribution notice. No share-alike or non-commercial data.

## Architecture

DistilBERT-base-uncased, 15 sigmoid outputs (14 scopes + `none`), `BCEWithLogitsLoss`
with per-label `pos_weight`, max_length 64. Artifact = `head.json` (labels, thresholds,
manifest, metrics) + `model/` (HF-standard).