AbstractPhil commited on
Commit
0c8f923
·
verified ·
1 Parent(s): 22eb16f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +73 -18
README.md CHANGED
@@ -42,30 +42,80 @@ of the 768 directions are actually used.
42
  | all-MiniLM-L6-v2 (ref) | 22.7M | .8203 | .7758 | +.023 | 94.3 |
43
 
44
  **It edges every teacher it was distilled from**, at 13% of their combined
45
- parameters, having never seen a similarity label. It does **not** reach
46
- `all-MiniLM-L6-v2`, which was contrastively trained on 1B+ curated pairs --
47
- a different comparison class.
 
48
 
49
  **Isotropy is the mechanism.** Mean-pooled BERT-family embeddings sit in a narrow
50
  cone (self_cos .58-.98); this model reads **+.129**, and cosine discriminates far
51
  better in a space that is not collapsed.
52
 
53
- ## With an AMOE anchor (`amoe/`)
54
 
55
- The trunk is frozen; adapters are 1.6M params each. Anchors ship in this repo
56
- under `amoe/` and toggle **bit-exact** -- all anchors disabled reproduces the
57
- bare trunk exactly, so one artifact serves both.
58
 
59
- | config | STS-B rho | SICK-R rho |
60
- |---|---|---|
61
- | bare trunk | .5747 | .6526 |
62
- | + `equiv` anchor (all-nli) | .7254 | **.7550** |
63
- | + `simplify` anchor (wiki/altlex/compression) | .7400 | .7075 |
64
- | + **2-anchor dispatch (MOE)** | **.7524** | .7380 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
 
66
- The two anchors are complementary along the task axis, and the router separated
67
- them: mean `|w/z|` moved from .310/.380 (blend) to .645/.223 (specialize) over
68
- 800 keys-only steps. See [amoe-lora](https://github.com/AbstractEyes/amoe-lora).
 
 
 
69
 
70
  ## How it was built
71
 
@@ -90,7 +140,12 @@ them: mean `|w/z|` moved from .310/.380 (blend) to .645/.223 (specialize) over
90
  - **10 of 66 source chunks lacked ModernBERT**, so 54 chunks (~27M rows) were
91
  used. No 4-expert fallback: that would change the target definition mid-dataset.
92
  - Trained on image captions. Expect caption-like text to be its strongest domain.
93
- - Single seed. No variance estimate.
 
 
 
 
 
94
 
95
  ## Output convention (differs from v1)
96
 
@@ -116,4 +171,4 @@ varied 0.2% across samples because a rotation round-trip carries no data.
116
  }
117
  ```
118
 
119
- MIT.
 
42
  | all-MiniLM-L6-v2 (ref) | 22.7M | .8203 | .7758 | +.023 | 94.3 |
43
 
44
  **It edges every teacher it was distilled from**, at 13% of their combined
45
+ parameters, having never seen a similarity label. The bare trunk does not reach
46
+ `all-MiniLM-L6-v2` (1B+ curated pairs, a different comparison class); with the
47
+ 3-arm AMOE collective below, STS-B closes to **.7684 vs .8203** and SICK-R to
48
+ **.7391 vs .7758**.
49
 
50
  **Isotropy is the mechanism.** Mean-pooled BERT-family embeddings sit in a narrow
51
  cone (self_cos .58-.98); this model reads **+.129**, and cosine discriminates far
52
  better in a space that is not collapsed.
53
 
54
+ ## With AMOE arms (`amoe/`)
55
 
56
+ The trunk is frozen; each anchor is 1.6M params across 12 sites. Anchors toggle
57
+ **bit-exact** -- all disabled reproduces the bare trunk exactly -- so one
58
+ artifact serves both the unsupervised baseline and the adapted model.
59
 
60
+ Best measured roster: **`equiv` + `simplify` + `paraphrase`** under a trained
61
+ dispatch, across the full 8-task STS suite:
62
+
63
+ | config | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
64
+ |---|---|---|---|---|---|---|---|---|---|
65
+ | bare trunk | .5747 | .6526 | .5051 | .5995 | .5452 | .7136 | .6776 | .5933 | .6077 |
66
+ | best single member | .7208 | .7269 | .6275 | .6988 | .6423 | .7783 | .7107 | .5845 | .6862 |
67
+ | **3-arm collective** | **.7684** | **.7391** | **.6682** | **.7557** | **.6921** | **.8055** | **.7626** | **.6382** | **.7287** |
68
+
69
+ **Eight for eight over every single member**, +.1210 mean over the trunk, from
70
+ 1.6M x 3 adapter params on a model that never moves. Note the BIOSSES column:
71
+ the strongest member alone reads .5845, *below* the bare trunk -- the mixture
72
+ turns a liability into a gain.
73
+
74
+ Solo anchor scores (trained alone, not masked inside a dispatch):
75
+
76
+ | anchor | trained on | relation | STS-B | SICK-R |
77
+ |---|---|---|---|---|
78
+ | `equiv` | all-nli | equivalence | .7254 | **.7550** |
79
+ | `simplify` | simple-wiki + altlex + sentence-compression | compression | **.7400** | .7075 |
80
+ | `paraphrase` | quora-duplicates | question paraphrase | see below | see below |
81
+
82
+ ### How many arms, and which
83
+
84
+ Arms do **not** add capacity -- they split a roughly conserved amplitude budget.
85
+ `w_k/z = sinh(u_k) / SUM_j cosh(u_j)` sums over *all* anchors, so measured sums
86
+ run .850 at A=2 to .757 at A=6 while each arm's share collapses. And sharper
87
+ routing does not rescue it: tau .10/.05/.02 gave STS-B .7520 / .7514 / .7468
88
+ while key separation *rose*. **The value is graded blending, not selection.**
89
+
90
+ So every candidate arm is judged against an **untrained arm at the same arm
91
+ count**, not against the smaller base. An empty arm gains +.0045 here -- real
92
+ generic capacity. Greedy forward selection, bar = the measured seed spread .0054:
93
+
94
+ | candidate (A=3) | STS-B | vs empty arm | verdict |
95
+ |---|---|---|---|
96
+ | `paraphrase` (quora) | .7684 | **+.0146** | **kept** |
97
+ | *untrained control* | .7538 | -- | baseline |
98
+ | `lexical` (WordNet term/gloss) | .7518 | -.0020 | rejected |
99
+ | `topical` (specter citations) | .7508 | -.0030 | rejected |
100
+
101
+ Every A=4 candidate -- including the control -- came in below .7684. Selection
102
+ stopped at three arms. Zero starvation alarms; per-block routing spread .348, so
103
+ the 12 dispatches learned different decisions rather than 12 copies of one.
104
+
105
+ ### Arm library
106
+
107
+ `amoe/arms/` holds one canonical copy of every anchor with `ARMS.json`
108
+ (provenance, training spec, solo scores, capacity verdict). **Resolve from
109
+ there; do not retrain what is already listed.** The campaign folders
110
+ (`amoe/{sts,sts-combo,moe,moe-v2,collective}`) remain as the records behind the
111
+ published numbers.
112
 
113
+ Scope worth stating: these anchors are solo-trained and always-on, which the
114
+ [aleph line](https://huggingface.co/blog/AbstractPhil/aleph-differentiation-ft3)
115
+ identifies as *blend* regime -- the dispatch blends them rather than containing
116
+ them. That is survivable here only because every arm serves one task. On a
117
+ multi-task trunk, expect the cross-task interference that record documents.
118
+ See [amoe-lora](https://github.com/AbstractEyes/amoe-lora).
119
 
120
  ## How it was built
121
 
 
140
  - **10 of 66 source chunks lacked ModernBERT**, so 54 chunks (~27M rows) were
141
  used. No 4-expert fallback: that would change the target definition mid-dataset.
142
  - Trained on image captions. Expect caption-like text to be its strongest domain.
143
+ - **The trunk is single-seed.** The AMOE results are 2-seed with a measured
144
+ spread of .003-.005, against margins of +.012 to +.019.
145
+ - `amoe/sts-combo` stopped at step 500 of a planned 4000; its card and config
146
+ describe the full run. `amoe/moe-v2` retrained its anchors instead of reusing
147
+ them, so the moe-vs-moe-v2 comparison moved two variables, not one. Both are
148
+ recorded rather than quietly corrected.
149
 
150
  ## Output convention (differs from v1)
151
 
 
171
  }
172
  ```
173
 
174
+ MIT.