amoe/sts-combo/README.md

---
license: mit
tags: [amoe, adapter, sentence-similarity, captionbert, aleph]
base_model: AbstractPhil/captionbert-8192-v2
library_name: amoe-lora
---

# captionbert-8192-v2 :: AMOE sentence-similarity anchor

An [amoe-lora](https://github.com/AbstractEyes/amoe-lora) aleph anchor trained on
the **frozen** `captionbert-8192-v2` trunk. The trunk never moves; only the
adapters train.

**This anchor is inert without its trunk.** It rewrites the residual stream of
specific blocks of this model and means nothing anywhere else, which is why it
ships here rather than as a standalone adapter repo.

## Why it exists

The trunk is an unsupervised consensus distillation of five BERT-family teachers.
Measured at release: the consensus target uses **28.7 of 768** directions, the
trunk uses **102.9 in domain** but only **~33 on STS-B**, so the extra structure
it built on CC12M captions does not transfer. This anchor asks whether
supervision can add transferable directions the unsupervised consensus never had.
The gauge is the **effective-rank delta with anchors ON vs OFF**, not the
similarity score alone.

## Toggle law

All anchors disabled == the base trunk, bit-exact in fp32 (asserted at train
time, not assumed). One artifact, both models:

```python
import amoe, torch
# trunk: see the parent repo for CaptionEncoder
h = amoe.attach(trunk, "amoe/sts-combo/captionbert-v2-sts-anchor-combo.anchor.pt",
                binding=CaptionEncoderBinding(d=512))
emb_supervised   = trunk(input_ids, attention_mask)
with h.only():                      # or set enabled=False on the wrapped blocks
    emb_unsupervised = trunk(input_ids, attention_mask)
base = h.detach()                   # bit-exact or raises
```

The `.pt` anchor layout is `{block}.{param}` (the `blocks.{site}.{param}`
form in the amoe README is the *safetensors* layout — a different serializer).

## Results

Measured at step 500 (`metrics.json`):

| config | STS-B rho | SICK-R rho |
|---|---|---|
| bare trunk | .5747 | .6526 |
| **combo anchor** | **.7344** | **.7454** |

SICK-R is never trained on and is the honest transfer read; the combo pack lifts
it +.0928 without ever seeing it.

The folder ships **one** anchor -- `captionbert-v2-sts-anchor-combo.anchor.pt`.
The solo-source anchor it is compared against lives in `amoe/sts/`.

## Trunk

These anchors expect **`checkpoints/best_model.pt`**, which is what the repo-root
`model.safetensors` carries (verified tensor-for-tensor, max absolute delta 0.0).
Anchors are trunk-bound and do not port to the sibling `-B` trunk -- measured
2026-08-02, ported anchors lose 31% of their gain there.

## Training

Frozen trunk, adapters only. MultipleNegativesRankingLoss on
a combo pack (all-nli + simple-wiki + altlex + sentence-compression)
with in-batch negatives and hard negatives where the source provides them.
Pure Adam wd=0 (`amoe.laws.make_optimizer`), fp32 / TF32 off
(`amoe.laws.pin_precision`). Config in `config.json`.