card: image first, opening cut to one line, noise removed
Browse files
README.md
CHANGED
|
@@ -17,118 +17,86 @@ datasets:
|
|
| 17 |
|
| 18 |
# Horos
|
| 19 |
|
| 20 |
-
**
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
on-device, multi-label scope router for [Topos](https://github.com/dialoguesai/topos)
|
| 24 |
-
that stands at the boundary of your data and names which region a question belongs to β
|
| 25 |
-
or refuses to let it pass.
|
| 26 |
-
Given a free-text question, it predicts which UMA personal-data **scopes** the question
|
| 27 |
-
touches (e.g. `health:read`, `schedule:read`), predicts an explicit **`none`** class for
|
| 28 |
-
questions that touch no personal data, or signals **uncertainty** so the caller can
|
| 29 |
-
escalate to an LLM. It runs *before* the permission gate β the raw question is the one
|
| 30 |
-
input the pipeline sees pre-grant, which is why this model exists: so that step never
|
| 31 |
-
has to leave the device.
|
| 32 |
|
| 33 |
-
|
| 34 |
-
still permission-checked downstream. A node without this model escalates to an LLM more
|
| 35 |
-
often; it does not become less safe.
|
| 36 |
|
| 37 |
-
|
|
|
|
|
|
|
| 38 |
|
| 39 |
-
|
|
|
|
| 40 |
|
| 41 |
-
##
|
| 42 |
|
| 43 |
-
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
```
|
| 46 |
labels β₯ tau_high, none low, nothing in band β ACT on the scope set (1..N scopes)
|
| 47 |
-
high none, nothing else β
|
| 48 |
-
any label in [tau_low, tau_high) β
|
| 49 |
-
nothing anywhere, none included β
|
| 50 |
```
|
| 51 |
|
| 52 |
-
|
| 53 |
-
classification task other than Topos scope routing (its 14 scope labels + `none` are the
|
| 54 |
-
Topos scope registry, versioned; the loader refuses the artifact if the live registry
|
| 55 |
-
has drifted).
|
| 56 |
|
| 57 |
-
|
|
|
|
|
|
|
| 58 |
|
| 59 |
-
|
| 60 |
-
disjoint from training, threshold 0.70 selected on a pre-training grouped holdout β
|
| 61 |
-
never on this benchmark.
|
| 62 |
-
|
| 63 |
-
**Do not anchor these numbers against standard intent benchmarks.** This benchmark is
|
| 64 |
-
deliberately adversarial: every positive uses phrasing templates the model never saw,
|
| 65 |
-
gold is multi-label, and 24% of cases are near-miss negatives (third-party probes,
|
| 66 |
-
mechanism/definition traps) built to punish keyword matching. For scale on the *same*
|
| 67 |
-
benchmark: **mistral:7b, a 4.4 GB LLM, scores 0.495 macro-F1** β this 268 MB model beats
|
| 68 |
-
it while running every turn on-device.
|
| 69 |
-
|
| 70 |
-
| metric | value | mistral:7b, same benchmark |
|
| 71 |
|---|---|---|
|
| 72 |
| macro-F1 | **0.512** | 0.495 |
|
| 73 |
| exact set match | **0.496** | 0.243 |
|
| 74 |
| negatives abstained | **0.984** | 0.126 |
|
| 75 |
-
| single
|
| 76 |
-
| dead rate
|
| 77 |
-
| disjoint rate
|
| 78 |
| per-scope recall β₯ 0.60 | 2 / 14 | 9 / 14 |
|
| 79 |
|
| 80 |
-
###
|
| 81 |
-
|
| 82 |
-
Composed per-case: the head's verdict stands on act/abstain; escalated turns take the
|
| 83 |
-
LLM's actual answer on those exact cases.
|
| 84 |
|
| 85 |
-
| arm | macro-F1 | exact | neg-abstain | wrong-scope
|
| 86 |
|---|---|---|---|---|---|
|
| 87 |
-
|
|
| 88 |
-
| **
|
| 89 |
-
|
|
| 90 |
-
| **
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
(0.
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
a
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
*
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
| schema-grounded synthetic (Topos) | internal/synthetic | 1,927 | positives |
|
| 118 |
-
| compound conjunctions of the above | internal/synthetic | 1,772 | multi-scope positives |
|
| 119 |
-
| negative twins + concept negatives | internal/synthetic | 1,465 | boundary negatives |
|
| 120 |
-
| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | 3,500 | real-register `none` + cue-mapped positives |
|
| 121 |
-
| [CLINC150 OOS](https://github.com/clinc/oos-eval) | CC BY 3.0 | 800 | out-of-scope `none` |
|
| 122 |
-
|
| 123 |
-
This model card is the attribution notice CC BY requires. No share-alike or
|
| 124 |
-
non-commercial data was used; the Topos loader enforces that class of licence refusal
|
| 125 |
-
at load time.
|
| 126 |
|
| 127 |
## Architecture
|
| 128 |
|
| 129 |
-
DistilBERT-base-uncased, 15
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
`model/` (HF-standard). Load through `topos.query.scope_head.load_head`, which validates
|
| 133 |
-
labels against the live scope registry and the manifest against the licence policy
|
| 134 |
-
before any weight is touched.
|
|
|
|
| 17 |
|
| 18 |
# Horos
|
| 19 |
|
| 20 |
+
**The boundary stone for personal data.** A 268 MB on-device router for
|
| 21 |
+
[Topos](https://github.com/dialoguesai/topos): free text in β the data scopes it touches,
|
| 22 |
+
an explicit `none`, or a hand-off to an LLM when it isn't sure.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
+

|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
*α½
ΟΞΏΟ β the stones Athenians planted at property lines: "whose land begins here." Same
|
| 27 |
+
job, for your data. Horos runs before the permission gate, so the raw question never
|
| 28 |
+
leaves the device.*
|
| 29 |
|
| 30 |
+
**Router, not authorizer.** Every scope it emits is still permission-checked downstream.
|
| 31 |
+
A node without Horos escalates more often; it does not become less safe.
|
| 32 |
|
| 33 |
+
## Use
|
| 34 |
|
| 35 |
+
The `scope` role in a Topos model pack. Load through `topos.query.scope_head.load_head`
|
| 36 |
+
β it validates the label set against the live scope registry and the training manifest
|
| 37 |
+
against the licence policy before touching a weight. Not intended for any task besides
|
| 38 |
+
Topos scope routing, and not intended to run without the escalation path.
|
| 39 |
|
| 40 |
```
|
| 41 |
labels β₯ tau_high, none low, nothing in band β ACT on the scope set (1..N scopes)
|
| 42 |
+
high none, nothing else β abstain: no personal data
|
| 43 |
+
any label in [tau_low, tau_high) β escalate to the pack's LLM (ambiguity)
|
| 44 |
+
nothing anywhere, none included β escalate to the pack's LLM (ignorance)
|
| 45 |
```
|
| 46 |
|
| 47 |
+
## Numbers
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
+
Benchmark: `role_classify_8` β 1,071 cases, template-disjoint from training, multi-label
|
| 50 |
+
gold, 24% adversarial negatives. It is built to be hard: don't anchor against CLINC-style
|
| 51 |
+
intent scores. The right column is the same benchmark's LLM baseline.
|
| 52 |
|
| 53 |
+
| metric | Horos | mistral:7b (4.4 GB) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|---|---|---|
|
| 55 |
| macro-F1 | **0.512** | 0.495 |
|
| 56 |
| exact set match | **0.496** | 0.243 |
|
| 57 |
| negatives abstained | **0.984** | 0.126 |
|
| 58 |
+
| single / multi-gold recall | 0.388 / 0.425 | β |
|
| 59 |
+
| dead rate | 0.171 | β |
|
| 60 |
+
| disjoint rate | 0.217 | β |
|
| 61 |
| per-scope recall β₯ 0.60 | 2 / 14 | 9 / 14 |
|
| 62 |
|
| 63 |
+
### With its LLM escalation (composed per-case)
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
+
| arm | macro-F1 | exact | neg-abstain | wrong-scope | LLM sees |
|
| 66 |
|---|---|---|---|---|---|
|
| 67 |
+
| mistral:7b alone | 0.495 | 0.243 | 0.126 | 0.237 | 100% |
|
| 68 |
+
| **Horos + mistral:7b** | **0.550** | **0.524** | **0.972** | **0.143** | **16.4%** |
|
| 69 |
+
| llama3.2 (2 GB) alone | 0.376 | 0.261 | 0.510 | 0.204 | 100% |
|
| 70 |
+
| **Horos + llama3.2** | **0.500** | **0.508** | **0.976** | 0.146 | 16.4% |
|
| 71 |
+
|
| 72 |
+
A 2 GB machine running the hybrid (0.500) beats a 4.4 GB machine running LLM-only
|
| 73 |
+
(0.495): Horos is identical on every hardware tier, so only the escalated ~1/6th of
|
| 74 |
+
traffic degrades with weaker hardware.
|
| 75 |
+
|
| 76 |
+
## Limitations
|
| 77 |
+
|
| 78 |
+
* **Confident-none swallowing.** 27% of benchmark positives get a confident `none` β
|
| 79 |
+
silently, no escalation. The dominant defect; mechanism is template familiarity.
|
| 80 |
+
Treat a `none` on a plausibly-personal question with suspicion.
|
| 81 |
+
* **12 of 14 scopes are under the 0.60 recall floor.** This artifact has not cleared its
|
| 82 |
+
promotion gate; it fronts an LLM in shadow/advisory postures only.
|
| 83 |
+
* All numbers are synthetic-benchmark. Real-traffic behaviour is being measured in
|
| 84 |
+
shadow mode.
|
| 85 |
+
* English only. **No user data, ever** β the loader refuses artifacts whose manifest
|
| 86 |
+
says otherwise.
|
| 87 |
+
|
| 88 |
+
## Training data
|
| 89 |
+
|
| 90 |
+
| source | licence | rows |
|
| 91 |
+
|---|---|---|
|
| 92 |
+
| schema-grounded synthetic + compounds + boundary negatives (Topos) | internal | 5,164 |
|
| 93 |
+
| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | 3,500 |
|
| 94 |
+
| [CLINC150 OOS](https://github.com/clinc/oos-eval) | CC BY 3.0 | 800 |
|
| 95 |
+
|
| 96 |
+
This card is the CC BY attribution notice. No share-alike or non-commercial data.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
|
| 98 |
## Architecture
|
| 99 |
|
| 100 |
+
DistilBERT-base-uncased, 15 sigmoid outputs (14 scopes + `none`), `BCEWithLogitsLoss`
|
| 101 |
+
with per-label `pos_weight`, max_length 64. Artifact = `head.json` (labels, thresholds,
|
| 102 |
+
manifest, metrics) + `model/` (HF-standard).
|
|
|
|
|
|
|
|
|