---
license: mit
language: [en]
library_name: transformers
pipeline_tag: feature-extraction
tags: [sentence-similarity, feature-extraction, consensus-distillation, geometric-deep-learning, amoe]
datasets: [AbstractPhil/conceptual-captions-12m-webdataset-berts]
base_model: [google-bert/bert-base-uncased, answerdotai/ModernBERT-base, FacebookAI/roberta-base, albert/albert-base-v2, distilbert/distilbert-base-uncased]
---

# captionbert-8192-b

A **58.3M** standalone sentence encoder distilled from the geometric **consensus**
of five BERT-family teachers. No expert models at inference: tokenizer + this
model, 768-d L2-normalized output.

12 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. **0.53x bert-base.**

This is the **complete-corpus** build: all 66 CC12M chunks, 31.9M rows. Its
sibling [`captionbert-8192-v2`](https://huggingface.co/AbstractPhil/captionbert-8192-v2)
trained on 54 chunks because ModernBERT was missing from 10 of them; those were
repaired and gate-verified before this run.

```python
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2-B", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

emb = model.encode(["a cat on a windowsill", "a feline by the window"])   # (2, 768)
(emb[0] @ emb[1]).item()

model.attach_amoe()          # this repo's NATIVE arms -- see the warning below
emb = model.encode(["a cat on a windowsill"])
```

## Benchmark

| model | params | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
|---|---|---|---|---|---|---|---|---|---|---|
| bert-base | 109.5M | 0.4729 | 0.5865 | 0.3087 | 0.5988 | 0.4773 | 0.6029 | 0.6373 | 0.5469 | 0.5289 |
| ModernBERT-base | 149.0M | 0.4215 | 0.5479 | 0.3527 | 0.4247 | 0.3795 | 0.5349 | 0.4174 | 0.5630 | 0.4552 |
| roberta-base | 124.6M | 0.5436 | 0.6296 | 0.3211 | 0.5631 | 0.4522 | 0.6134 | 0.6198 | 0.5777 | 0.5401 |
| albert-base-v2 | 11.7M | 0.4784 | 0.5364 | 0.3101 | 0.4831 | 0.3809 | 0.5542 | 0.5491 | 0.4863 | 0.4723 |
| distilbert | 66.4M | 0.5717 | 0.6424 | 0.4344 | 0.6490 | 0.5410 | 0.6663 | 0.6854 | 0.5162 | 0.5883 |
| **captionbert-8192-b** | 58.3M | **0.5752** | **0.6548** | **0.5012** | **0.6037** | **0.5470** | **0.7146** | **0.6782** | **0.5500** | **0.6031** |
| **captionbert-8192-b + arms** | 63.2M | **0.7675** | **0.7374** | **0.6706** | **0.7381** | **0.6945** | **0.8109** | **0.7695** | **0.6472** | **0.7295** |
| captionbert-8192-v2 | 58.3M | 0.5747 | 0.6526 | 0.5051 | 0.5995 | 0.5452 | 0.7136 | 0.6776 | 0.5933 | 0.6077 |
| all-MiniLM-L6-v2 | 22.7M | 0.8203 | 0.7758 | 0.7237 | 0.8058 | 0.7559 | 0.8539 | 0.7899 | 0.8144 | 0.7925 |

All ten models measured in **one harness**, same eight tasks, **mean-pooled and
L2-normalized**, no task tuning. Spearman correlation; `mean` is the unweighted
average over the eight.

`all-MiniLM-L6-v2` was contrastively trained on 1B+ curated sentence pairs. It is
listed for scale, not as a peer -- nothing here saw a similarity label.

**The trunk beats every teacher it was distilled from**, and the best of them
(distilbert, .5883) by +.0194 -- at **13% of their combined 461M parameters**,
having never seen a similarity label. The margin comes mostly from STS12, where
every teacher collapses to .31-.43 and the trunk holds .50.

**With arms it clears the best teacher by +.14** and closes to within **.063** of
a model trained on a billion curated pairs.

Mean-pooled BERT-family encoders are known-weak sentence encoders -- that is the
reason Sentence-BERT exists -- so beating them is an efficiency result rather
than a state-of-the-art one. The MiniLM row is in the table to keep that honest.

### Geometry

| model | self_cos | erank |
|---|---|---|
| bert-base | +0.6071 | 32.8 |
| ModernBERT-base | +0.9001 | 26.1 |
| roberta-base | +0.9594 | 19.8 |
| albert-base-v2 | +0.7473 | 20.9 |
| distilbert | +0.6920 | 31.1 |
| **captionbert-8192-b** | +0.1411 | 36.1 |
| **captionbert-8192-b + arms** | +0.0984 | 55.5 |
| captionbert-8192-v2 | +0.1396 | 36.6 |
| all-MiniLM-L6-v2 | +0.0251 | 86.7 |

`self_cos` is the isotropy gauge: the mean cosine between unrelated sentences.
Mean-pooled BERT-family embeddings sit in a narrow cone (+.61 to +.96), where
cosine cannot discriminate. `erank` is the participation ratio -- how many of the
768 directions carry variance.

Both track capability almost perfectly across all ten models, and **isotropy is
the mechanism**: no isotropy objective appears anywhere in the training stack.
The arms then lift erank 36.6 -> 57.6, the first evidence in this line that
adaptation *adds* usable directions rather than only rotating them.

## More data bought nothing (and that is the finding)

`-b` trained on **19% more rows for 19% more steps** than `-v2`. Head to head:

| | v2 (54ch, 26.9M) | b (66ch, 31.9M) | delta |
|---|---|---|---|
| 8-task mean, bare | .6077 | .6031 | -.0046 |
| 7 tasks excluding BIOSSES | -- | -- | **+.0009** |
| erank (STS-B) | 36.6 | 36.1 | -0.5 |
| self_cos (STS-B) | +.1396 | +.1411 | +.0015 |
| 8-task mean, native arms | .7287 | **.7295** | **+.0008** |

The entire -.0046 comes from BIOSSES, which is 100 rows -- a 0.4-sigma move.
Everything else is a dead heat.

**The ceiling is TEACHER AGREEMENT, not corpus size.** The consensus target uses
**28.7 of 768 directions**: five BERT-family encoders only agree on ~29, and no
amount of the same distribution raises that. The trunk reaches erank ~103 *in
domain* but ~36 out of it -- the structure it builds on captions does not
transfer. The next lever is heterogeneous teachers, measurable at the consensus
stage before a single training step.

## AMOE arms are TRUNK-BOUND -- use this repo's

Three 1.6M-parameter anchors on the frozen trunk, under a trained dispatch.
Anchors toggle **bit-exact**, so one artifact serves both the unsupervised
baseline and the adapted model.

| mask | STS-B | SICK-R | mean (8 tasks) |
|---|---|---|---|
| OFF (bare trunk) | .5752 | .6548 | .6031 |
| `equiv` only | .7219 | .7200 | .6842 |
| `simplify` only | .5995 | .6603 | .6254 |
| `paraphrase` only | .6137 | .6612 | .6295 |
| **all three** | **.7675** | **.7374** | **.7295** |

An `-only` row is that arm **as damped by the dispatch** -- masking never
renormalizes, so it reads lower than the same anchor trained alone.

**Do not attach `captionbert-8192-v2`'s arms to this trunk.** Measured:

| configuration | mean |
|---|---|
| v2 arms on v2 | .7287 |
| v2 arms on **-b** | .6863 |
| + re-aligned routing keys | .6987 |
| **-b native anchors** | **.7295** |

Transferring the arms costs **31% of their gain**. Re-training only the 1,536
routing keys recovers 29% of that; retraining the anchors recovers all of it.
**71% of the loss is in the anchors themselves.**

These two trunks are indistinguishable on eight STS tasks and on geometry, yet
1.6M adapter parameters tell them apart -- adapters read the residual stream and
the task gauges read the pooled output, and the stream carries trunk identity
the output does not. Budget one anchor set per trunk (~18 min).

`attach_amoe()` resolves this repo's own arms by default. Files are under
`amoe/b-collective/`. See [amoe-lora](https://github.com/AbstractEyes/amoe-lora).

## How it was built

1. Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d).
2. One global **whitened Procrustes** map per teacher into `bert-base`'s frame,
   fit on a stratified random sample and **reported out-of-sample** (worst arm
   retains 95% of its in-sample R@1 at 1,833x chance).
3. Consensus = normalized centroid of the aligned teachers, per chunk.
4. Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the
   consensus. Pure Adam, no weight decay. 31.9M rows, 62,312 steps at batch
   2048, ~6.4 h on one RTX 6000 Pro.

The alignment maps in `maps/` are **the same maps v2 used** -- refitting them
would put the consensus targets in a different frame with no signal in the loss.

## Known limits

- **Consensus rank ~28.7 of 768.** The model's ceiling, and a property of
  teacher agreement rather than of this model.
- **Alignment quality varies by teacher.** Out-of-sample cosine into the bert
  frame: distil .625, roberta .372, albert .331, modern .327 -- the ordering
  tracks architectural distance from bert-base.
- **Single seed.** The AMOE results carry a measured seed spread of .003-.005;
  the trunk does not have one.
- Trained on image captions; expect caption-like text to be its strongest domain.
- BIOSSES is 100 rows. Treat any single-task delta there as noise.

## Files

```
model.safetensors            the trunk, HF format
config.json                  AutoModel config (auto_map -> modeling_captionbert)
modeling_captionbert.py      CaptionBertV2Model + attach_amoe/detach_amoe
checkpoints/                 training checkpoints (final_model.pt is the ship)
maps/                        alignment maps -- SHARED with v2, do not refit
amoe/b-collective/           native anchors + dispatch + metrics
```

## Output convention

| field | shape | |
|---|---|---|
| `last_hidden_state` | (B, L, 512) | token states |
| `pooler_output` | (B, 768) | **the embedding**, L2-normalized |
| `embedding` | (B, 768) | alias |

`geolip-captionbert-8192` (v1) returned the pooled embedding as
`last_hidden_state`. If porting v1 code, use `pooler_output`.


## deep-arm/ — long-context binding attachment (optional, detachable)

The base trunk's attribute binding is semantically alive to ~256 tokens
(its trained position range) and collapses beyond it — measured with a
minimal-pair battery ("a red cube on a blue sphere" vs swaps, ratio of
own-attribute to other-attribute state movement at the noun positions;
1.0 = chance). `deep-arm/` restores deep binding **without touching the
trunk**: 4.98M trainable parameters distilled from
`allenai/longformer-base-4096` token states (span-resampled across
tokenizers, mapped 768→512 by a whitened-Procrustes fit, out-of-sample
cos .501 / retrieval R@1 .849 vs a dead shuffled null).

**Construction**: (1) position rows 256+ re-initialized by mod-256
tiling of the trained 0–255 table, then trained (rows 0–255 frozen);
(2) one gated 16-slot relay adapter per encoder block (gates open
monotonically with depth, .35–.51 after training); (3) per-token cosine
distillation to the mapped Longformer states over long caption
documents, deep-weighted.

**Binding at depth** (battery ratios, before → after; both alignment
phases of the tiling shown):

| payload depth | before | after |
|---|---|---|
| 10 | 1.59 / 2.01 | 2.69 / 2.27 |
| 480 (tile edge) | 1.17 / 1.13 | 1.63 / 3.50 |
| 1024 (aligned) | 3.12 / 2.92 | 2.66 / 3.96 |
| 1248 (tile edge) | 1.22 / 1.12 | 3.18 / 3.80 |
| 2048 (aligned) | 3.31 / 2.73 | 4.83 / 13.1 |
| 2288 (tile edge) | 1.13 / 0.98 | 1.63 / 1.58 |

(The tiled init alone restores the aligned depths; the trained deep
rows repair the tile edges in a near-to-far wave; the relays amplify
retro-binding wherever gradient reaches. The 13.1 cell is flagged
pending an absolute-distance decomposition.)

**The honest cost**: with the attachment ENGAGED, short-input capability
drops .6031 → .5655 on the 8-task STS mean and shallow isotropy degrades
(self_cos +.003 → +.288) — the Longformer-mapped frame is anisotropic.
The attachment is therefore a **length-conditional mode**: adapters are
σ-gated wrappers and rows 0–255 are untouched, so with the wrappers
removed (or gated off) short-input behavior is bit-identical to the
stock trunk. Engage for inputs past ~256 tokens; run stock below.

**Use**: load the trunk as above; from `deep-arm/deep1_arm_s0.pt` copy
`pos_emb.weight`, wrap each `encoder.layers[i]` with its `block{i}.*`
relay (a residual adapter applied to the block output), or skip both to
recover the stock model exactly. `deep-arm/deep1_results.json` carries
the full battery and the fit report.

## Citation

```bibtex
@misc{abstractphil2026captionbertb,
  title  = {captionbert-8192-b: consensus distillation on the complete CC12M corpus},
  author = {AbstractPhil},
  year   = {2026},
  url    = {https://huggingface.co/AbstractPhil/captionbert-8192-v2-B}
}
```

MIT.