amoe/sts/README.md

metadata

license: mit
tags:
  - amoe
  - adapter
  - sentence-similarity
  - captionbert
  - aleph
base_model: AbstractPhil/captionbert-8192-v2
library_name: amoe-lora

captionbert-8192-v2 :: AMOE sentence-similarity anchor

An amoe-lora aleph anchor trained on the frozen captionbert-8192-v2 trunk. The trunk never moves; only the adapters train.

This anchor is inert without its trunk. It rewrites the residual stream of specific blocks of this model and means nothing anywhere else, which is why it ships here rather than as a standalone adapter repo.

Why it exists

The trunk is an unsupervised consensus distillation of five BERT-family teachers. Measured at release: the consensus target uses 28.7 of 768 directions, the trunk uses 102.9 in domain but only ~33 on STS-B, so the extra structure it built on CC12M captions does not transfer. This anchor asks whether supervision can add transferable directions the unsupervised consensus never had. The gauge is the effective-rank delta with anchors ON vs OFF, not the similarity score alone.

Toggle law

All anchors disabled == the base trunk, bit-exact in fp32 (asserted at train time, not assumed). One artifact, both models:

import amoe, torch
# trunk: see the parent repo for CaptionEncoder
h = amoe.attach(trunk, "amoe/sts/captionbert-v2-sts-anchor.anchor.pt", binding=CaptionEncoderBinding(d=512))
emb_supervised   = trunk(input_ids, attention_mask)
with h.only():                      # or set enabled=False on the wrapped blocks
    emb_unsupervised = trunk(input_ids, attention_mask)
base = h.detach()                   # bit-exact or raises

The .pt anchor layout is {block}.{param} (the blocks.{site}.{param} form in the amoe README is the safetensors layout — a different serializer).

Results

See metrics.json in this folder. Report STS-B / SICK-R spearman with anchors ON and OFF, plus effective rank for each. SICK-R is never trained on and is the honest transfer read.

Training

Frozen trunk, adapters only. MultipleNegativesRankingLoss on sentence-transformers/all-nli triplets with in-batch + hard negatives. Pure Adam wd=0 (amoe.laws.make_optimizer), fp32 / TF32 off (amoe.laws.pin_precision). Config in config.json.