license: mit
tags:
- amoe
- adapter
- sentence-similarity
- captionbert
- aleph
base_model: AbstractPhil/captionbert-8192-v2
library_name: amoe-lora
captionbert-8192-v2 :: AMOE sentence-similarity anchor
An amoe-lora aleph anchor trained on
the frozen captionbert-8192-v2 trunk. The trunk never moves; only the
adapters train.
This anchor is inert without its trunk. It rewrites the residual stream of specific blocks of this model and means nothing anywhere else, which is why it ships here rather than as a standalone adapter repo.
Why it exists
The trunk is an unsupervised consensus distillation of five BERT-family teachers. Measured at release: the consensus target uses 28.7 of 768 directions, the trunk uses 102.9 in domain but only ~33 on STS-B, so the extra structure it built on CC12M captions does not transfer. This anchor asks whether supervision can add transferable directions the unsupervised consensus never had. The gauge is the effective-rank delta with anchors ON vs OFF, not the similarity score alone.
Toggle law
All anchors disabled == the base trunk, bit-exact in fp32 (asserted at train time, not assumed). One artifact, both models:
import amoe, torch
# trunk: see the parent repo for CaptionEncoder
h = amoe.attach(trunk, "amoe/sts-combo/captionbert-v2-sts-anchor-combo.anchor.pt",
binding=CaptionEncoderBinding(d=512))
emb_supervised = trunk(input_ids, attention_mask)
with h.only(): # or set enabled=False on the wrapped blocks
emb_unsupervised = trunk(input_ids, attention_mask)
base = h.detach() # bit-exact or raises
The .pt anchor layout is {block}.{param} (the blocks.{site}.{param}
form in the amoe README is the safetensors layout — a different serializer).
Results
Measured at step 500 (metrics.json):
| config | STS-B rho | SICK-R rho |
|---|---|---|
| bare trunk | .5747 | .6526 |
| combo anchor | .7344 | .7454 |
SICK-R is never trained on and is the honest transfer read; the combo pack lifts it +.0928 without ever seeing it.
The folder ships one anchor -- captionbert-v2-sts-anchor-combo.anchor.pt.
The solo-source anchor it is compared against lives in amoe/sts/.
Trunk
These anchors expect checkpoints/best_model.pt, which is what the repo-root
model.safetensors carries (verified tensor-for-tensor, max absolute delta 0.0).
Anchors are trunk-bound and do not port to the sibling -B trunk -- measured
2026-08-02, ported anchors lose 31% of their gain there.
Training
Frozen trunk, adapters only. MultipleNegativesRankingLoss on
a combo pack (all-nli + simple-wiki + altlex + sentence-compression)
with in-batch negatives and hard negatives where the source provides them.
Pure Adam wd=0 (amoe.laws.make_optimizer), fp32 / TF32 off
(amoe.laws.pin_precision). Config in config.json.