File size: 2,301 Bytes
2ebd035 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | ---
license: mit
tags: [amoe, adapter, sentence-similarity, captionbert, aleph]
base_model: AbstractPhil/captionbert-8192-v2
library_name: amoe-lora
---
# captionbert-8192-v2 :: AMOE sentence-similarity anchor
An [amoe-lora](https://github.com/AbstractEyes/amoe-lora) aleph anchor trained on
the **frozen** `captionbert-8192-v2` trunk. The trunk never moves; only the
adapters train.
**This anchor is inert without its trunk.** It rewrites the residual stream of
specific blocks of this model and means nothing anywhere else, which is why it
ships here rather than as a standalone adapter repo.
## Why it exists
The trunk is an unsupervised consensus distillation of five BERT-family teachers.
Measured at release: the consensus target uses **28.7 of 768** directions, the
trunk uses **102.9 in domain** but only **~33 on STS-B**, so the extra structure
it built on CC12M captions does not transfer. This anchor asks whether
supervision can add transferable directions the unsupervised consensus never had.
The gauge is the **effective-rank delta with anchors ON vs OFF**, not the
similarity score alone.
## Toggle law
All anchors disabled == the base trunk, bit-exact in fp32 (asserted at train
time, not assumed). One artifact, both models:
```python
import amoe, torch
# trunk: see the parent repo for CaptionEncoder
h = amoe.attach(trunk, "amoe/sts/captionbert-v2-sts-anchor.anchor.pt", binding=CaptionEncoderBinding(d=512))
emb_supervised = trunk(input_ids, attention_mask)
with h.only(): # or set enabled=False on the wrapped blocks
emb_unsupervised = trunk(input_ids, attention_mask)
base = h.detach() # bit-exact or raises
```
The `.pt` anchor layout is `{block}.{param}` (the `blocks.{site}.{param}`
form in the amoe README is the *safetensors* layout — a different serializer).
## Results
See `metrics.json` in this folder. Report STS-B / SICK-R spearman with anchors
ON and OFF, plus effective rank for each. SICK-R is never trained on and is the
honest transfer read.
## Training
Frozen trunk, adapters only. MultipleNegativesRankingLoss on
`sentence-transformers/all-nli` triplets with in-batch + hard negatives.
Pure Adam wd=0 (`amoe.laws.make_optimizer`), fp32 / TF32 off
(`amoe.laws.pin_precision`). Config in `config.json`.
|