amoe/sts/README.md

---
license: mit
tags: [amoe, adapter, sentence-similarity, captionbert, aleph]
base_model: AbstractPhil/captionbert-8192-v2
library_name: amoe-lora
---

# captionbert-8192-v2 :: AMOE sentence-similarity anchor

An [amoe-lora](https://github.com/AbstractEyes/amoe-lora) aleph anchor trained on
the **frozen** `captionbert-8192-v2` trunk. The trunk never moves; only the
adapters train.

**This anchor is inert without its trunk.** It rewrites the residual stream of
specific blocks of this model and means nothing anywhere else, which is why it
ships here rather than as a standalone adapter repo.

## Why it exists

The trunk is an unsupervised consensus distillation of five BERT-family teachers.
Measured at release: the consensus target uses **28.7 of 768** directions, the
trunk uses **102.9 in domain** but only **~33 on STS-B**, so the extra structure
it built on CC12M captions does not transfer. This anchor asks whether
supervision can add transferable directions the unsupervised consensus never had.
The gauge is the **effective-rank delta with anchors ON vs OFF**, not the
similarity score alone.

## Toggle law

All anchors disabled == the base trunk, bit-exact in fp32 (asserted at train
time, not assumed). One artifact, both models:

```python
import amoe, torch
# trunk: see the parent repo for CaptionEncoder
h = amoe.attach(trunk, "amoe/sts/captionbert-v2-sts-anchor.anchor.pt", binding=CaptionEncoderBinding(d=512))
emb_supervised   = trunk(input_ids, attention_mask)
with h.only():                      # or set enabled=False on the wrapped blocks
    emb_unsupervised = trunk(input_ids, attention_mask)
base = h.detach()                   # bit-exact or raises
```

The `.pt` anchor layout is `{block}.{param}` (the `blocks.{site}.{param}`
form in the amoe README is the *safetensors* layout — a different serializer).

## Results

See `metrics.json` in this folder. Report STS-B / SICK-R spearman with anchors
ON and OFF, plus effective rank for each. SICK-R is never trained on and is the
honest transfer read.

## Training

Frozen trunk, adapters only. MultipleNegativesRankingLoss on
`sentence-transformers/all-nli` triplets with in-batch + hard negatives.
Pure Adam wd=0 (`amoe.laws.make_optimizer`), fp32 / TF32 off
(`amoe.laws.pin_precision`). Config in `config.json`.