Feature Extraction
Transformers
TensorBoard
Safetensors
English
captionbert_v2
sentence-similarity
consensus-distillation
geometric-deep-learning
amoe
custom_code
Instructions to use AbstractPhil/captionbert-8192-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AbstractPhil/captionbert-8192-v2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="AbstractPhil/captionbert-8192-v2", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -42,30 +42,80 @@ of the 768 directions are actually used.
|
|
| 42 |
| all-MiniLM-L6-v2 (ref) | 22.7M | .8203 | .7758 | +.023 | 94.3 |
|
| 43 |
|
| 44 |
**It edges every teacher it was distilled from**, at 13% of their combined
|
| 45 |
-
parameters, having never seen a similarity label.
|
| 46 |
-
`all-MiniLM-L6-v2`
|
| 47 |
-
|
|
|
|
| 48 |
|
| 49 |
**Isotropy is the mechanism.** Mean-pooled BERT-family embeddings sit in a narrow
|
| 50 |
cone (self_cos .58-.98); this model reads **+.129**, and cosine discriminates far
|
| 51 |
better in a space that is not collapsed.
|
| 52 |
|
| 53 |
-
## With
|
| 54 |
|
| 55 |
-
The trunk is frozen;
|
| 56 |
-
|
| 57 |
-
|
| 58 |
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
|
| 63 |
-
|
|
| 64 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
## How it was built
|
| 71 |
|
|
@@ -90,7 +140,12 @@ them: mean `|w/z|` moved from .310/.380 (blend) to .645/.223 (specialize) over
|
|
| 90 |
- **10 of 66 source chunks lacked ModernBERT**, so 54 chunks (~27M rows) were
|
| 91 |
used. No 4-expert fallback: that would change the target definition mid-dataset.
|
| 92 |
- Trained on image captions. Expect caption-like text to be its strongest domain.
|
| 93 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
|
| 95 |
## Output convention (differs from v1)
|
| 96 |
|
|
@@ -116,4 +171,4 @@ varied 0.2% across samples because a rotation round-trip carries no data.
|
|
| 116 |
}
|
| 117 |
```
|
| 118 |
|
| 119 |
-
MIT.
|
|
|
|
| 42 |
| all-MiniLM-L6-v2 (ref) | 22.7M | .8203 | .7758 | +.023 | 94.3 |
|
| 43 |
|
| 44 |
**It edges every teacher it was distilled from**, at 13% of their combined
|
| 45 |
+
parameters, having never seen a similarity label. The bare trunk does not reach
|
| 46 |
+
`all-MiniLM-L6-v2` (1B+ curated pairs, a different comparison class); with the
|
| 47 |
+
3-arm AMOE collective below, STS-B closes to **.7684 vs .8203** and SICK-R to
|
| 48 |
+
**.7391 vs .7758**.
|
| 49 |
|
| 50 |
**Isotropy is the mechanism.** Mean-pooled BERT-family embeddings sit in a narrow
|
| 51 |
cone (self_cos .58-.98); this model reads **+.129**, and cosine discriminates far
|
| 52 |
better in a space that is not collapsed.
|
| 53 |
|
| 54 |
+
## With AMOE arms (`amoe/`)
|
| 55 |
|
| 56 |
+
The trunk is frozen; each anchor is 1.6M params across 12 sites. Anchors toggle
|
| 57 |
+
**bit-exact** -- all disabled reproduces the bare trunk exactly -- so one
|
| 58 |
+
artifact serves both the unsupervised baseline and the adapted model.
|
| 59 |
|
| 60 |
+
Best measured roster: **`equiv` + `simplify` + `paraphrase`** under a trained
|
| 61 |
+
dispatch, across the full 8-task STS suite:
|
| 62 |
+
|
| 63 |
+
| config | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
|
| 64 |
+
|---|---|---|---|---|---|---|---|---|---|
|
| 65 |
+
| bare trunk | .5747 | .6526 | .5051 | .5995 | .5452 | .7136 | .6776 | .5933 | .6077 |
|
| 66 |
+
| best single member | .7208 | .7269 | .6275 | .6988 | .6423 | .7783 | .7107 | .5845 | .6862 |
|
| 67 |
+
| **3-arm collective** | **.7684** | **.7391** | **.6682** | **.7557** | **.6921** | **.8055** | **.7626** | **.6382** | **.7287** |
|
| 68 |
+
|
| 69 |
+
**Eight for eight over every single member**, +.1210 mean over the trunk, from
|
| 70 |
+
1.6M x 3 adapter params on a model that never moves. Note the BIOSSES column:
|
| 71 |
+
the strongest member alone reads .5845, *below* the bare trunk -- the mixture
|
| 72 |
+
turns a liability into a gain.
|
| 73 |
+
|
| 74 |
+
Solo anchor scores (trained alone, not masked inside a dispatch):
|
| 75 |
+
|
| 76 |
+
| anchor | trained on | relation | STS-B | SICK-R |
|
| 77 |
+
|---|---|---|---|---|
|
| 78 |
+
| `equiv` | all-nli | equivalence | .7254 | **.7550** |
|
| 79 |
+
| `simplify` | simple-wiki + altlex + sentence-compression | compression | **.7400** | .7075 |
|
| 80 |
+
| `paraphrase` | quora-duplicates | question paraphrase | see below | see below |
|
| 81 |
+
|
| 82 |
+
### How many arms, and which
|
| 83 |
+
|
| 84 |
+
Arms do **not** add capacity -- they split a roughly conserved amplitude budget.
|
| 85 |
+
`w_k/z = sinh(u_k) / SUM_j cosh(u_j)` sums over *all* anchors, so measured sums
|
| 86 |
+
run .850 at A=2 to .757 at A=6 while each arm's share collapses. And sharper
|
| 87 |
+
routing does not rescue it: tau .10/.05/.02 gave STS-B .7520 / .7514 / .7468
|
| 88 |
+
while key separation *rose*. **The value is graded blending, not selection.**
|
| 89 |
+
|
| 90 |
+
So every candidate arm is judged against an **untrained arm at the same arm
|
| 91 |
+
count**, not against the smaller base. An empty arm gains +.0045 here -- real
|
| 92 |
+
generic capacity. Greedy forward selection, bar = the measured seed spread .0054:
|
| 93 |
+
|
| 94 |
+
| candidate (A=3) | STS-B | vs empty arm | verdict |
|
| 95 |
+
|---|---|---|---|
|
| 96 |
+
| `paraphrase` (quora) | .7684 | **+.0146** | **kept** |
|
| 97 |
+
| *untrained control* | .7538 | -- | baseline |
|
| 98 |
+
| `lexical` (WordNet term/gloss) | .7518 | -.0020 | rejected |
|
| 99 |
+
| `topical` (specter citations) | .7508 | -.0030 | rejected |
|
| 100 |
+
|
| 101 |
+
Every A=4 candidate -- including the control -- came in below .7684. Selection
|
| 102 |
+
stopped at three arms. Zero starvation alarms; per-block routing spread .348, so
|
| 103 |
+
the 12 dispatches learned different decisions rather than 12 copies of one.
|
| 104 |
+
|
| 105 |
+
### Arm library
|
| 106 |
+
|
| 107 |
+
`amoe/arms/` holds one canonical copy of every anchor with `ARMS.json`
|
| 108 |
+
(provenance, training spec, solo scores, capacity verdict). **Resolve from
|
| 109 |
+
there; do not retrain what is already listed.** The campaign folders
|
| 110 |
+
(`amoe/{sts,sts-combo,moe,moe-v2,collective}`) remain as the records behind the
|
| 111 |
+
published numbers.
|
| 112 |
|
| 113 |
+
Scope worth stating: these anchors are solo-trained and always-on, which the
|
| 114 |
+
[aleph line](https://huggingface.co/blog/AbstractPhil/aleph-differentiation-ft3)
|
| 115 |
+
identifies as *blend* regime -- the dispatch blends them rather than containing
|
| 116 |
+
them. That is survivable here only because every arm serves one task. On a
|
| 117 |
+
multi-task trunk, expect the cross-task interference that record documents.
|
| 118 |
+
See [amoe-lora](https://github.com/AbstractEyes/amoe-lora).
|
| 119 |
|
| 120 |
## How it was built
|
| 121 |
|
|
|
|
| 140 |
- **10 of 66 source chunks lacked ModernBERT**, so 54 chunks (~27M rows) were
|
| 141 |
used. No 4-expert fallback: that would change the target definition mid-dataset.
|
| 142 |
- Trained on image captions. Expect caption-like text to be its strongest domain.
|
| 143 |
+
- **The trunk is single-seed.** The AMOE results are 2-seed with a measured
|
| 144 |
+
spread of .003-.005, against margins of +.012 to +.019.
|
| 145 |
+
- `amoe/sts-combo` stopped at step 500 of a planned 4000; its card and config
|
| 146 |
+
describe the full run. `amoe/moe-v2` retrained its anchors instead of reusing
|
| 147 |
+
them, so the moe-vs-moe-v2 comparison moved two variables, not one. Both are
|
| 148 |
+
recorded rather than quietly corrected.
|
| 149 |
|
| 150 |
## Output convention (differs from v1)
|
| 151 |
|
|
|
|
| 171 |
}
|
| 172 |
```
|
| 173 |
|
| 174 |
+
MIT.
|