README.md

---
license: mit
language: [en]
library_name: transformers
pipeline_tag: feature-extraction
tags: [sentence-similarity, feature-extraction, consensus-distillation, geometric-deep-learning, amoe]
datasets: [AbstractPhil/conceptual-captions-12m-webdataset-berts]
base_model: [google-bert/bert-base-uncased, answerdotai/ModernBERT-base, FacebookAI/roberta-base, albert/albert-base-v2, distilbert/distilbert-base-uncased]
---

# captionbert-8192-v2

A **58.3M** standalone sentence encoder distilled from the geometric **consensus**
of five BERT-family teachers. No expert models at inference: tokenizer + this
model, 768-d L2-normalized output.

12 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. **0.53x bert-base.**

```python
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

emb = model.encode(["a cat on a windowsill", "a feline by the window"])   # (2, 768)
(emb[0] @ emb[1]).item()
```

## Benchmark

| model | params | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
|---|---|---|---|---|---|---|---|---|---|---|
| bert-base | 109.5M | 0.4729 | 0.5865 | 0.3087 | 0.5988 | 0.4773 | 0.6029 | 0.6373 | 0.5469 | 0.5289 |
| ModernBERT-base | 149.0M | 0.4215 | 0.5479 | 0.3527 | 0.4247 | 0.3795 | 0.5349 | 0.4174 | 0.5630 | 0.4552 |
| roberta-base | 124.6M | 0.5436 | 0.6296 | 0.3211 | 0.5631 | 0.4522 | 0.6134 | 0.6198 | 0.5777 | 0.5401 |
| albert-base-v2 | 11.7M | 0.4784 | 0.5364 | 0.3101 | 0.4831 | 0.3809 | 0.5542 | 0.5491 | 0.4863 | 0.4723 |
| distilbert | 66.4M | 0.5717 | 0.6424 | 0.4344 | 0.6490 | 0.5410 | 0.6663 | 0.6854 | 0.5162 | 0.5883 |
| **captionbert-8192-v2** | 58.3M | **0.5747** | **0.6526** | **0.5051** | **0.5995** | **0.5452** | **0.7136** | **0.6776** | **0.5933** | **0.6077** |
| **captionbert-8192-v2 + arms** | 63.2M | **0.7684** | **0.7391** | **0.6682** | **0.7557** | **0.6921** | **0.8055** | **0.7626** | **0.6382** | **0.7287** |
| captionbert-8192-b | 58.3M | 0.5752 | 0.6548 | 0.5012 | 0.6037 | 0.5470 | 0.7146 | 0.6782 | 0.5500 | 0.6031 |
| all-MiniLM-L6-v2 | 22.7M | 0.8203 | 0.7758 | 0.7237 | 0.8058 | 0.7559 | 0.8539 | 0.7899 | 0.8144 | 0.7925 |

All ten models measured in **one harness**, same eight tasks, **mean-pooled and
L2-normalized**, no task tuning. Spearman correlation; `mean` is the unweighted
average over the eight.

`all-MiniLM-L6-v2` was contrastively trained on 1B+ curated sentence pairs. It is
listed for scale, not as a peer -- nothing here saw a similarity label.

**The trunk beats every teacher it was distilled from**, and the best of them
(distilbert, .5883) by +.0194 -- at **13% of their combined 461M parameters**,
having never seen a similarity label. The margin comes mostly from STS12, where
every teacher collapses to .31-.43 and the trunk holds .50.

**With arms it clears the best teacher by +.14** and closes to within **.063** of
a model trained on a billion curated pairs.

Mean-pooled BERT-family encoders are known-weak sentence encoders -- that is the
reason Sentence-BERT exists -- so beating them is an efficiency result rather
than a state-of-the-art one. The MiniLM row is in the table to keep that honest.

### Geometry

| model | self_cos | erank |
|---|---|---|
| bert-base | +0.6071 | 32.8 |
| ModernBERT-base | +0.9001 | 26.1 |
| roberta-base | +0.9594 | 19.8 |
| albert-base-v2 | +0.7473 | 20.9 |
| distilbert | +0.6920 | 31.1 |
| **captionbert-8192-v2** | +0.1396 | 36.6 |
| **captionbert-8192-v2 + arms** | +0.0964 | 57.6 |
| captionbert-8192-b | +0.1411 | 36.1 |
| all-MiniLM-L6-v2 | +0.0251 | 86.7 |

`self_cos` is the isotropy gauge: the mean cosine between unrelated sentences.
Mean-pooled BERT-family embeddings sit in a narrow cone (+.61 to +.96), where
cosine cannot discriminate. `erank` is the participation ratio -- how many of the
768 directions carry variance.

Both track capability almost perfectly across all ten models, and **isotropy is
the mechanism**: no isotropy objective appears anywhere in the training stack.
The arms then lift erank 36.6 -> 57.6, the first evidence in this line that
adaptation *adds* usable directions rather than only rotating them.

## With AMOE arms (`amoe/`)

The trunk is frozen; each anchor is 1.6M params across 12 sites. Anchors toggle
**bit-exact** -- all disabled reproduces the bare trunk exactly -- so one
artifact serves both the unsupervised baseline and the adapted model.

Best measured roster: **`equiv` + `simplify` + `paraphrase`** under a trained
dispatch, across the full 8-task STS suite:

| config | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
|---|---|---|---|---|---|---|---|---|---|
| bare trunk | .5747 | .6526 | .5051 | .5995 | .5452 | .7136 | .6776 | .5933 | .6077 |
| best single member | .7208 | .7269 | .6275 | .6988 | .6423 | .7783 | .7107 | .5845 | .6862 |
| **3-arm collective** | **.7684** | **.7391** | **.6682** | **.7557** | **.6921** | **.8055** | **.7626** | **.6382** | **.7287** |

**Eight for eight over every single member**, +.1210 mean over the trunk, from
1.6M x 3 adapter params on a model that never moves. Note the BIOSSES column:
the strongest member alone reads .5845, *below* the bare trunk -- the mixture
turns a liability into a gain.

Solo anchor scores (trained alone, not masked inside a dispatch):

| anchor | trained on | relation | STS-B | SICK-R |
|---|---|---|---|---|
| `equiv` | all-nli | equivalence | .7254 | **.7550** |
| `simplify` | simple-wiki + altlex + sentence-compression | compression | **.7400** | .7075 |
| `paraphrase` | quora-duplicates | question paraphrase | see below | see below |

### How many arms, and which

Arms do **not** add capacity -- they split a roughly conserved amplitude budget.
`w_k/z = sinh(u_k) / SUM_j cosh(u_j)` sums over *all* anchors, so measured sums
run .850 at A=2 to .757 at A=6 while each arm's share collapses. And sharper
routing does not rescue it: tau .10/.05/.02 gave STS-B .7520 / .7514 / .7468
while key separation *rose*. **The value is graded blending, not selection.**

So every candidate arm is judged against an **untrained arm at the same arm
count**, not against the smaller base. An empty arm gains +.0045 here -- real
generic capacity. Greedy forward selection, bar = the measured seed spread .0054:

| candidate (A=3) | STS-B | vs empty arm | verdict |
|---|---|---|---|
| `paraphrase` (quora) | .7684 | **+.0146** | **kept** |
| *untrained control* | .7538 | -- | baseline |
| `lexical` (WordNet term/gloss) | .7518 | -.0020 | rejected |
| `topical` (specter citations) | .7508 | -.0030 | rejected |

Every A=4 candidate -- including the control -- came in below .7684. Selection
stopped at three arms. Zero starvation alarms; per-block routing spread .348, so
the 12 dispatches learned different decisions rather than 12 copies of one.

### Loading arms

Arms attach through the same `AutoModel` object. `amoe-lora` is imported lazily,
so the base model still loads on a machine that has never heard of it.

```python
from transformers import AutoModel
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2",
                                  trust_remote_code=True)

model.attach_amoe()                       # the shipped 3-arm collective
emb = model.encode(["a cat on a windowsill"])

with model.amoe_off():                    # the unsupervised baseline
    base = model.encode(["a cat on a windowsill"])

model.set_amoe(["equiv"])                 # one arm, inside the dispatch
model.detach_amoe()                       # bit-exact restore, asserted
```

`attach_amoe()` with no arguments reads the roster **and its order** from the
dispatch checkpoint -- routing keys are per-arm and positional, so order is not
cosmetic. Both `attach_amoe` and `detach_amoe` assert the toggle law and raise
rather than return a silently-wrong model.

`pip install git+https://github.com/AbstractEyes/amoe-lora` for the arm methods.

### Arms do not transfer between trunks

These anchors were trained against **this** trunk's residual stream. Measured
against the sibling [`captionbert-8192-b`](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B)
(same architecture, complete 66-chunk corpus):

| configuration | 8-task mean |
|---|---|
| v2 arms on v2 | .7287 |
| v2 arms on **-b** | .6863 |
| + re-aligned routing keys | .6987 |
| -b's own anchors on -b | .7295 |

Transferring costs **31% of the gain**; re-training only the 1,536 routing keys
recovers 29% of that, so **71% of the loss is in the anchors themselves**.

The two trunks are indistinguishable on all eight tasks (+.0009 mean excluding
the 100-row BIOSSES) and on geometry, yet 1.6M adapter parameters tell them
apart -- adapters read the residual stream, the task gauges read the pooled
output, and the stream carries trunk identity the output does not. Budget one
anchor set per trunk (~18 min).

### Arm library

`amoe/arms/` holds one canonical copy of every anchor with `ARMS.json`
(provenance, training spec, solo scores, capacity verdict). **Resolve from
there; do not retrain what is already listed.** The campaign folders
(`amoe/{sts,sts-combo,moe,moe-v2,collective}`) remain as the records behind the
published numbers.

Scope worth stating: these anchors are solo-trained and always-on, which the
[aleph line](https://huggingface.co/blog/AbstractPhil/aleph-differentiation-ft3)
identifies as *blend* regime -- the dispatch blends them rather than containing
them. That is survivable here only because every arm serves one task. On a
multi-task trunk, expect the cross-task interference that record documents.
See [amoe-lora](https://github.com/AbstractEyes/amoe-lora).

## How it was built

1. Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d).
2. One global **whitened Procrustes** map per teacher into `bert-base`'s frame,
   fit on a stratified random sample and **reported out-of-sample**.
3. Consensus = normalized centroid of the aligned teachers, per chunk.
4. Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the
   consensus. Pure Adam, no weight decay. 26.9M rows, 52,548 steps at batch
   2048, ~5.4 h on one RTX 6000 Pro.

## Known limits -- read before using

- **Consensus rank is ~28.7 of 768.** Five BERT-family teachers only agree on
  about 29 directions. The student uses ~103 in domain but falls back to ~33 on
  out-of-domain text: **the structure it builds on captions does not transfer.**
  This is the model's ceiling and it is a property of the consensus, not the
  student.
- **Alignment quality varies by teacher.** Out-of-sample cosine to the bert
  frame: distil .625, roberta .372, albert .331, modern .327. The ordering
  tracks architectural distance from bert-base.
- **10 of 66 source chunks lacked ModernBERT**, so 54 chunks (~27M rows) were
  used. No 4-expert fallback: that would change the target definition mid-dataset.
- Trained on image captions. Expect caption-like text to be its strongest domain.
- **A sibling trained on the complete 66-chunk corpus** ([`-b`](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B))
  scores within .0046 of this one bare and .0008 with arms. 19% more data bought
  nothing: the ceiling is teacher agreement (28.7 of 768 shared directions), not
  corpus size.
- **The trunk is single-seed.** The AMOE results are 2-seed with a measured
  spread of .003-.005, against margins of +.012 to +.019.
- `amoe/sts-combo` stopped at step 500 of a planned 4000; its card and config
  describe the full run. `amoe/moe-v2` retrained its anchors instead of reusing
  them, so the moe-vs-moe-v2 comparison moved two variables, not one. Both are
  recorded rather than quietly corrected.

## Output convention (differs from v1)

| field | shape | |
|---|---|---|
| `last_hidden_state` | (B, L, 512) | token states |
| `pooler_output` | (B, 768) | **the embedding**, L2-normalized |
| `embedding` | (B, 768) | alias |

`geolip-captionbert-8192` (v1) returned the pooled embedding as
`last_hidden_state`. If porting v1 code, use `pooler_output`. v1 also shipped an
`AlignmentBank`; v2 does not -- measured on v1, its expert-consistency features
varied 0.2% across samples because a rotation round-trip carries no data.

## Citation

```bibtex
@misc{abstractphil2026captionbertv2,
  title  = {captionbert-8192-v2: consensus distillation at CC12M scale},
  author = {AbstractPhil},
  year   = {2026},
  url    = {https://huggingface.co/AbstractPhil/captionbert-8192-v2}
}
```

MIT.