jonny commited on
Commit
257f5bd
Β·
verified Β·
1 Parent(s): ac7f810

card: image first, opening cut to one line, noise removed

Browse files
Files changed (1) hide show
  1. README.md +60 -92
README.md CHANGED
@@ -17,118 +17,86 @@ datasets:
17
 
18
  # Horos
19
 
20
- **ὅρος** β€” the inscribed boundary stones of ancient Athens. A horos stood at the edge of
21
- a plot and announced whose land began there; some carried mortgage terms, some marked
22
- sacred precincts nobody could enter. This model is a horos for personal data: an
23
- on-device, multi-label scope router for [Topos](https://github.com/dialoguesai/topos)
24
- that stands at the boundary of your data and names which region a question belongs to β€”
25
- or refuses to let it pass.
26
- Given a free-text question, it predicts which UMA personal-data **scopes** the question
27
- touches (e.g. `health:read`, `schedule:read`), predicts an explicit **`none`** class for
28
- questions that touch no personal data, or signals **uncertainty** so the caller can
29
- escalate to an LLM. It runs *before* the permission gate β€” the raw question is the one
30
- input the pipeline sees pre-grant, which is why this model exists: so that step never
31
- has to leave the device.
32
 
33
- **This model is a router, not an authorization mechanism.** Every scope it emits is
34
- still permission-checked downstream. A node without this model escalates to an LLM more
35
- often; it does not become less safe.
36
 
37
- ## How it routes
 
 
38
 
39
- ![How Horos routes a question: act on confident scope sets (1..N), abstain on confident none, escalate ambiguity and ignorance to the pack's LLM](routing.png)
 
40
 
41
- ## Intended use
42
 
43
- Deployed as the `scope` role in a Topos model pack, behind the four-branch ladder above:
 
 
 
44
 
45
  ```
46
  labels β‰₯ tau_high, none low, nothing in band β†’ ACT on the scope set (1..N scopes)
47
- high none, nothing else β†’ confident abstain (no personal data)
48
- any label in [tau_low, tau_high) β†’ ESCALATE to the pack's LLM (ambiguity)
49
- nothing anywhere, none included β†’ ESCALATE to the pack's LLM (ignorance)
50
  ```
51
 
52
- It is **not** intended to be used without the escalation path, and not intended for any
53
- classification task other than Topos scope routing (its 14 scope labels + `none` are the
54
- Topos scope registry, versioned; the loader refuses the artifact if the live registry
55
- has drifted).
56
 
57
- ## Numbers, with the method
 
 
58
 
59
- All measured on `role_classify_8` (1,071 cases, 818 positive / 253 negative), template-
60
- disjoint from training, threshold 0.70 selected on a pre-training grouped holdout β€”
61
- never on this benchmark.
62
-
63
- **Do not anchor these numbers against standard intent benchmarks.** This benchmark is
64
- deliberately adversarial: every positive uses phrasing templates the model never saw,
65
- gold is multi-label, and 24% of cases are near-miss negatives (third-party probes,
66
- mechanism/definition traps) built to punish keyword matching. For scale on the *same*
67
- benchmark: **mistral:7b, a 4.4 GB LLM, scores 0.495 macro-F1** β€” this 268 MB model beats
68
- it while running every turn on-device.
69
-
70
- | metric | value | mistral:7b, same benchmark |
71
  |---|---|---|
72
  | macro-F1 | **0.512** | 0.495 |
73
  | exact set match | **0.496** | 0.243 |
74
  | negatives abstained | **0.984** | 0.126 |
75
- | single-gold vs multi-gold recall | 0.388 / 0.425 (gap βˆ’0.036) | β€” |
76
- | dead rate (no opinion incl. `none`) | 0.171 | β€” |
77
- | disjoint rate (acted, set shares nothing with gold) | 0.217 | β€” |
78
  | per-scope recall β‰₯ 0.60 | 2 / 14 | 9 / 14 |
79
 
80
- ### As the hybrid it is designed to front (head + LLM escalation)
81
-
82
- Composed per-case: the head's verdict stands on act/abstain; escalated turns take the
83
- LLM's actual answer on those exact cases.
84
 
85
- | arm | macro-F1 | exact | neg-abstain | wrong-scope/turns | LLM sees |
86
  |---|---|---|---|---|---|
87
- | LLM-only mistral:7b (4.4 GB) | 0.495 | 0.243 | 0.126 | 0.237 | 100% |
88
- | **hybrid: this model + mistral:7b** | **0.550** | **0.524** | **0.972** | **0.143** | **16.4%** |
89
- | LLM-only llama3.2 (2 GB) | 0.376 | 0.261 | 0.510 | 0.204 | 100% |
90
- | **hybrid: this model + llama3.2** | **0.500** | **0.508** | **0.976** | 0.146 | 16.4% |
91
-
92
- The result that matters for low-RAM machines: **a 2 GB-LLM machine running the hybrid
93
- (0.500) outperforms a 4.4 GB-LLM machine running LLM-only (0.495)** β€” the head is 268 MB
94
- and identical on every hardware tier, so only ~1/6th of traffic degrades with weaker
95
- hardware.
96
-
97
- ## Known limitations β€” read before relying on it
98
-
99
- * **Confident-none swallowing.** 222 of 818 benchmark positives (27%) are answered with
100
- a confident `none` β€” silently, with no escalation. This is the model's dominant defect.
101
- Measured mechanism: template familiarity (heldout phrasings fall to `none`), not class
102
- balance. Callers should treat a `none` on a plausibly-personal question with suspicion.
103
- * **Per-scope recall is far below production floors** (12 of 14 scopes under 0.60).
104
- This artifact has **not** cleared its promotion gate and fronts an LLM in shadow /
105
- advisory postures only.
106
- * All numbers above are **synthetic-benchmark** numbers. Real-traffic behaviour is being
107
- measured in shadow mode; median confidence on real language is substantially lower
108
- than on generated text.
109
- * English only. Trained entirely on synthetic templates + the public corpora below β€”
110
- **no user data, ever** (the Topos loader refuses artifacts whose manifest violates
111
- this).
112
-
113
- ## Training data & attribution
114
-
115
- | source | licence | rows | role |
116
- |---|---|---|---|
117
- | schema-grounded synthetic (Topos) | internal/synthetic | 1,927 | positives |
118
- | compound conjunctions of the above | internal/synthetic | 1,772 | multi-scope positives |
119
- | negative twins + concept negatives | internal/synthetic | 1,465 | boundary negatives |
120
- | [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | 3,500 | real-register `none` + cue-mapped positives |
121
- | [CLINC150 OOS](https://github.com/clinc/oos-eval) | CC BY 3.0 | 800 | out-of-scope `none` |
122
-
123
- This model card is the attribution notice CC BY requires. No share-alike or
124
- non-commercial data was used; the Topos loader enforces that class of licence refusal
125
- at load time.
126
 
127
  ## Architecture
128
 
129
- DistilBERT-base-uncased, 15-label multi-label head (14 scopes + `none`), sigmoid
130
- outputs, `BCEWithLogitsLoss` with per-label `pos_weight`, max_length 64. Artifact format
131
- `topos-scope-head-1`: `head.json` (labels, thresholds, corpus manifest, metrics) +
132
- `model/` (HF-standard). Load through `topos.query.scope_head.load_head`, which validates
133
- labels against the live scope registry and the manifest against the licence policy
134
- before any weight is touched.
 
17
 
18
  # Horos
19
 
20
+ **The boundary stone for personal data.** A 268 MB on-device router for
21
+ [Topos](https://github.com/dialoguesai/topos): free text in β†’ the data scopes it touches,
22
+ an explicit `none`, or a hand-off to an LLM when it isn't sure.
 
 
 
 
 
 
 
 
 
23
 
24
+ ![How Horos routes a question](routing.png)
 
 
25
 
26
+ *ὅρος β€” the stones Athenians planted at property lines: "whose land begins here." Same
27
+ job, for your data. Horos runs before the permission gate, so the raw question never
28
+ leaves the device.*
29
 
30
+ **Router, not authorizer.** Every scope it emits is still permission-checked downstream.
31
+ A node without Horos escalates more often; it does not become less safe.
32
 
33
+ ## Use
34
 
35
+ The `scope` role in a Topos model pack. Load through `topos.query.scope_head.load_head`
36
+ β€” it validates the label set against the live scope registry and the training manifest
37
+ against the licence policy before touching a weight. Not intended for any task besides
38
+ Topos scope routing, and not intended to run without the escalation path.
39
 
40
  ```
41
  labels β‰₯ tau_high, none low, nothing in band β†’ ACT on the scope set (1..N scopes)
42
+ high none, nothing else β†’ abstain: no personal data
43
+ any label in [tau_low, tau_high) β†’ escalate to the pack's LLM (ambiguity)
44
+ nothing anywhere, none included β†’ escalate to the pack's LLM (ignorance)
45
  ```
46
 
47
+ ## Numbers
 
 
 
48
 
49
+ Benchmark: `role_classify_8` β€” 1,071 cases, template-disjoint from training, multi-label
50
+ gold, 24% adversarial negatives. It is built to be hard: don't anchor against CLINC-style
51
+ intent scores. The right column is the same benchmark's LLM baseline.
52
 
53
+ | metric | Horos | mistral:7b (4.4 GB) |
 
 
 
 
 
 
 
 
 
 
 
54
  |---|---|---|
55
  | macro-F1 | **0.512** | 0.495 |
56
  | exact set match | **0.496** | 0.243 |
57
  | negatives abstained | **0.984** | 0.126 |
58
+ | single / multi-gold recall | 0.388 / 0.425 | β€” |
59
+ | dead rate | 0.171 | β€” |
60
+ | disjoint rate | 0.217 | β€” |
61
  | per-scope recall β‰₯ 0.60 | 2 / 14 | 9 / 14 |
62
 
63
+ ### With its LLM escalation (composed per-case)
 
 
 
64
 
65
+ | arm | macro-F1 | exact | neg-abstain | wrong-scope | LLM sees |
66
  |---|---|---|---|---|---|
67
+ | mistral:7b alone | 0.495 | 0.243 | 0.126 | 0.237 | 100% |
68
+ | **Horos + mistral:7b** | **0.550** | **0.524** | **0.972** | **0.143** | **16.4%** |
69
+ | llama3.2 (2 GB) alone | 0.376 | 0.261 | 0.510 | 0.204 | 100% |
70
+ | **Horos + llama3.2** | **0.500** | **0.508** | **0.976** | 0.146 | 16.4% |
71
+
72
+ A 2 GB machine running the hybrid (0.500) beats a 4.4 GB machine running LLM-only
73
+ (0.495): Horos is identical on every hardware tier, so only the escalated ~1/6th of
74
+ traffic degrades with weaker hardware.
75
+
76
+ ## Limitations
77
+
78
+ * **Confident-none swallowing.** 27% of benchmark positives get a confident `none` β€”
79
+ silently, no escalation. The dominant defect; mechanism is template familiarity.
80
+ Treat a `none` on a plausibly-personal question with suspicion.
81
+ * **12 of 14 scopes are under the 0.60 recall floor.** This artifact has not cleared its
82
+ promotion gate; it fronts an LLM in shadow/advisory postures only.
83
+ * All numbers are synthetic-benchmark. Real-traffic behaviour is being measured in
84
+ shadow mode.
85
+ * English only. **No user data, ever** β€” the loader refuses artifacts whose manifest
86
+ says otherwise.
87
+
88
+ ## Training data
89
+
90
+ | source | licence | rows |
91
+ |---|---|---|
92
+ | schema-grounded synthetic + compounds + boundary negatives (Topos) | internal | 5,164 |
93
+ | [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | 3,500 |
94
+ | [CLINC150 OOS](https://github.com/clinc/oos-eval) | CC BY 3.0 | 800 |
95
+
96
+ This card is the CC BY attribution notice. No share-alike or non-commercial data.
 
 
 
 
 
 
 
 
 
97
 
98
  ## Architecture
99
 
100
+ DistilBERT-base-uncased, 15 sigmoid outputs (14 scopes + `none`), `BCEWithLogitsLoss`
101
+ with per-label `pos_weight`, max_length 64. Artifact = `head.json` (labels, thresholds,
102
+ manifest, metrics) + `model/` (HF-standard).