--- license: mit language: [en] library_name: transformers pipeline_tag: feature-extraction tags: [sentence-similarity, feature-extraction, consensus-distillation, geometric-deep-learning, amoe] datasets: [AbstractPhil/conceptual-captions-12m-webdataset-berts] base_model: [google-bert/bert-base-uncased, answerdotai/ModernBERT-base, FacebookAI/roberta-base, albert/albert-base-v2, distilbert/distilbert-base-uncased] --- # captionbert-8192-v2 A **58.3M** standalone sentence encoder distilled from the geometric **consensus** of five BERT-family teachers. No expert models at inference: tokenizer + this model, 768-d L2-normalized output. 12 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. **0.53x bert-base.** ```python from transformers import AutoModel, AutoTokenizer model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True) tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") emb = model.encode(["a cat on a windowsill", "a feline by the window"]) # (2, 768) (emb[0] @ emb[1]).item() ``` ## Benchmark | model | params | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean | |---|---|---|---|---|---|---|---|---|---|---| | bert-base | 109.5M | 0.4729 | 0.5865 | 0.3087 | 0.5988 | 0.4773 | 0.6029 | 0.6373 | 0.5469 | 0.5289 | | ModernBERT-base | 149.0M | 0.4215 | 0.5479 | 0.3527 | 0.4247 | 0.3795 | 0.5349 | 0.4174 | 0.5630 | 0.4552 | | roberta-base | 124.6M | 0.5436 | 0.6296 | 0.3211 | 0.5631 | 0.4522 | 0.6134 | 0.6198 | 0.5777 | 0.5401 | | albert-base-v2 | 11.7M | 0.4784 | 0.5364 | 0.3101 | 0.4831 | 0.3809 | 0.5542 | 0.5491 | 0.4863 | 0.4723 | | distilbert | 66.4M | 0.5717 | 0.6424 | 0.4344 | 0.6490 | 0.5410 | 0.6663 | 0.6854 | 0.5162 | 0.5883 | | **captionbert-8192-v2** | 58.3M | **0.5747** | **0.6526** | **0.5051** | **0.5995** | **0.5452** | **0.7136** | **0.6776** | **0.5933** | **0.6077** | | **captionbert-8192-v2 + arms** | 63.2M | **0.7684** | **0.7391** | **0.6682** | **0.7557** | **0.6921** | **0.8055** | **0.7626** | **0.6382** | **0.7287** | | captionbert-8192-b | 58.3M | 0.5752 | 0.6548 | 0.5012 | 0.6037 | 0.5470 | 0.7146 | 0.6782 | 0.5500 | 0.6031 | | all-MiniLM-L6-v2 | 22.7M | 0.8203 | 0.7758 | 0.7237 | 0.8058 | 0.7559 | 0.8539 | 0.7899 | 0.8144 | 0.7925 | All ten models measured in **one harness**, same eight tasks, **mean-pooled and L2-normalized**, no task tuning. Spearman correlation; `mean` is the unweighted average over the eight. `all-MiniLM-L6-v2` was contrastively trained on 1B+ curated sentence pairs. It is listed for scale, not as a peer -- nothing here saw a similarity label. **The trunk beats every teacher it was distilled from**, and the best of them (distilbert, .5883) by +.0194 -- at **13% of their combined 461M parameters**, having never seen a similarity label. The margin comes mostly from STS12, where every teacher collapses to .31-.43 and the trunk holds .50. **With arms it clears the best teacher by +.14** and closes to within **.063** of a model trained on a billion curated pairs. Mean-pooled BERT-family encoders are known-weak sentence encoders -- that is the reason Sentence-BERT exists -- so beating them is an efficiency result rather than a state-of-the-art one. The MiniLM row is in the table to keep that honest. ### Geometry | model | self_cos | erank | |---|---|---| | bert-base | +0.6071 | 32.8 | | ModernBERT-base | +0.9001 | 26.1 | | roberta-base | +0.9594 | 19.8 | | albert-base-v2 | +0.7473 | 20.9 | | distilbert | +0.6920 | 31.1 | | **captionbert-8192-v2** | +0.1396 | 36.6 | | **captionbert-8192-v2 + arms** | +0.0964 | 57.6 | | captionbert-8192-b | +0.1411 | 36.1 | | all-MiniLM-L6-v2 | +0.0251 | 86.7 | `self_cos` is the isotropy gauge: the mean cosine between unrelated sentences. Mean-pooled BERT-family embeddings sit in a narrow cone (+.61 to +.96), where cosine cannot discriminate. `erank` is the participation ratio -- how many of the 768 directions carry variance. Both track capability almost perfectly across all ten models, and **isotropy is the mechanism**: no isotropy objective appears anywhere in the training stack. The arms then lift erank 36.6 -> 57.6, the first evidence in this line that adaptation *adds* usable directions rather than only rotating them. ## With AMOE arms (`amoe/`) The trunk is frozen; each anchor is 1.6M params across 12 sites. Anchors toggle **bit-exact** -- all disabled reproduces the bare trunk exactly -- so one artifact serves both the unsupervised baseline and the adapted model. Best measured roster: **`equiv` + `simplify` + `paraphrase`** under a trained dispatch, across the full 8-task STS suite: | config | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean | |---|---|---|---|---|---|---|---|---|---| | bare trunk | .5747 | .6526 | .5051 | .5995 | .5452 | .7136 | .6776 | .5933 | .6077 | | best single member | .7208 | .7269 | .6275 | .6988 | .6423 | .7783 | .7107 | .5845 | .6862 | | **3-arm collective** | **.7684** | **.7391** | **.6682** | **.7557** | **.6921** | **.8055** | **.7626** | **.6382** | **.7287** | **Eight for eight over every single member**, +.1210 mean over the trunk, from 1.6M x 3 adapter params on a model that never moves. Note the BIOSSES column: the strongest member alone reads .5845, *below* the bare trunk -- the mixture turns a liability into a gain. Solo anchor scores (trained alone, not masked inside a dispatch): | anchor | trained on | relation | STS-B | SICK-R | |---|---|---|---|---| | `equiv` | all-nli | equivalence | .7254 | **.7550** | | `simplify` | simple-wiki + altlex + sentence-compression | compression | **.7400** | .7075 | | `paraphrase` | quora-duplicates | question paraphrase | see below | see below | ### How many arms, and which Arms do **not** add capacity -- they split a roughly conserved amplitude budget. `w_k/z = sinh(u_k) / SUM_j cosh(u_j)` sums over *all* anchors, so measured sums run .850 at A=2 to .757 at A=6 while each arm's share collapses. And sharper routing does not rescue it: tau .10/.05/.02 gave STS-B .7520 / .7514 / .7468 while key separation *rose*. **The value is graded blending, not selection.** So every candidate arm is judged against an **untrained arm at the same arm count**, not against the smaller base. An empty arm gains +.0045 here -- real generic capacity. Greedy forward selection, bar = the measured seed spread .0054: | candidate (A=3) | STS-B | vs empty arm | verdict | |---|---|---|---| | `paraphrase` (quora) | .7684 | **+.0146** | **kept** | | *untrained control* | .7538 | -- | baseline | | `lexical` (WordNet term/gloss) | .7518 | -.0020 | rejected | | `topical` (specter citations) | .7508 | -.0030 | rejected | Every A=4 candidate -- including the control -- came in below .7684. Selection stopped at three arms. Zero starvation alarms; per-block routing spread .348, so the 12 dispatches learned different decisions rather than 12 copies of one. ### Loading arms Arms attach through the same `AutoModel` object. `amoe-lora` is imported lazily, so the base model still loads on a machine that has never heard of it. ```python from transformers import AutoModel model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True) model.attach_amoe() # the shipped 3-arm collective emb = model.encode(["a cat on a windowsill"]) with model.amoe_off(): # the unsupervised baseline base = model.encode(["a cat on a windowsill"]) model.set_amoe(["equiv"]) # one arm, inside the dispatch model.detach_amoe() # bit-exact restore, asserted ``` `attach_amoe()` with no arguments reads the roster **and its order** from the dispatch checkpoint -- routing keys are per-arm and positional, so order is not cosmetic. Both `attach_amoe` and `detach_amoe` assert the toggle law and raise rather than return a silently-wrong model. `pip install git+https://github.com/AbstractEyes/amoe-lora` for the arm methods. ### Arms do not transfer between trunks These anchors were trained against **this** trunk's residual stream. Measured against the sibling [`captionbert-8192-b`](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B) (same architecture, complete 66-chunk corpus): | configuration | 8-task mean | |---|---| | v2 arms on v2 | .7287 | | v2 arms on **-b** | .6863 | | + re-aligned routing keys | .6987 | | -b's own anchors on -b | .7295 | Transferring costs **31% of the gain**; re-training only the 1,536 routing keys recovers 29% of that, so **71% of the loss is in the anchors themselves**. The two trunks are indistinguishable on all eight tasks (+.0009 mean excluding the 100-row BIOSSES) and on geometry, yet 1.6M adapter parameters tell them apart -- adapters read the residual stream, the task gauges read the pooled output, and the stream carries trunk identity the output does not. Budget one anchor set per trunk (~18 min). ### Arm library `amoe/arms/` holds one canonical copy of every anchor with `ARMS.json` (provenance, training spec, solo scores, capacity verdict). **Resolve from there; do not retrain what is already listed.** The campaign folders (`amoe/{sts,sts-combo,moe,moe-v2,collective}`) remain as the records behind the published numbers. Scope worth stating: these anchors are solo-trained and always-on, which the [aleph line](https://huggingface.co/blog/AbstractPhil/aleph-differentiation-ft3) identifies as *blend* regime -- the dispatch blends them rather than containing them. That is survivable here only because every arm serves one task. On a multi-task trunk, expect the cross-task interference that record documents. See [amoe-lora](https://github.com/AbstractEyes/amoe-lora). ## How it was built 1. Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d). 2. One global **whitened Procrustes** map per teacher into `bert-base`'s frame, fit on a stratified random sample and **reported out-of-sample**. 3. Consensus = normalized centroid of the aligned teachers, per chunk. 4. Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the consensus. Pure Adam, no weight decay. 26.9M rows, 52,548 steps at batch 2048, ~5.4 h on one RTX 6000 Pro. ## Known limits -- read before using - **Consensus rank is ~28.7 of 768.** Five BERT-family teachers only agree on about 29 directions. The student uses ~103 in domain but falls back to ~33 on out-of-domain text: **the structure it builds on captions does not transfer.** This is the model's ceiling and it is a property of the consensus, not the student. - **Alignment quality varies by teacher.** Out-of-sample cosine to the bert frame: distil .625, roberta .372, albert .331, modern .327. The ordering tracks architectural distance from bert-base. - **10 of 66 source chunks lacked ModernBERT**, so 54 chunks (~27M rows) were used. No 4-expert fallback: that would change the target definition mid-dataset. - Trained on image captions. Expect caption-like text to be its strongest domain. - **A sibling trained on the complete 66-chunk corpus** ([`-b`](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B)) scores within .0046 of this one bare and .0008 with arms. 19% more data bought nothing: the ceiling is teacher agreement (28.7 of 768 shared directions), not corpus size. - **The trunk is single-seed.** The AMOE results are 2-seed with a measured spread of .003-.005, against margins of +.012 to +.019. - `amoe/sts-combo` stopped at step 500 of a planned 4000; its card and config describe the full run. `amoe/moe-v2` retrained its anchors instead of reusing them, so the moe-vs-moe-v2 comparison moved two variables, not one. Both are recorded rather than quietly corrected. ## Output convention (differs from v1) | field | shape | | |---|---|---| | `last_hidden_state` | (B, L, 512) | token states | | `pooler_output` | (B, 768) | **the embedding**, L2-normalized | | `embedding` | (B, 768) | alias | `geolip-captionbert-8192` (v1) returned the pooled embedding as `last_hidden_state`. If porting v1 code, use `pooler_output`. v1 also shipped an `AlignmentBank`; v2 does not -- measured on v1, its expert-consistency features varied 0.2% across samples because a rotation round-trip carries no data. ## Citation ```bibtex @misc{abstractphil2026captionbertv2, title = {captionbert-8192-v2: consensus distillation at CC12M scale}, author = {AbstractPhil}, year = {2026}, url = {https://huggingface.co/AbstractPhil/captionbert-8192-v2} } ``` MIT.