amoe/moe/README.md

---
license: mit
tags: [amoe, adapter, mixture-of-experts, sentence-similarity, captionbert, aleph]
base_model: AbstractPhil/captionbert-8192-v2
library_name: amoe-lora
---

# captionbert-8192-v2 :: AMOE 2-anchor mixture

Two [amoe-lora](https://github.com/AbstractEyes/amoe-lora) aleph anchors on the
**frozen** trunk, plus a trained dispatch over them. The trunk never moves.

| anchor | trained on | relation |
|---|---|---|
| `equiv` | all-nli triplets | semantic equivalence |
| `simplify` | simple-wiki + altlex + sentence-compression | simplification / compression |

## Results

| config | STS-B rho | SICK-R rho |
|---|---|---|
| bare trunk | .5747 | .6526 |
| `equiv` alone | .7254 | **.7550** |
| `simplify` alone | .7400 | .7075 |
| **2-anchor dispatch** | **.7524** | .7380 |

The two are complementary along the TASK axis -- `simplify` wins STS-B solo,
`equiv` wins SICK-R -- which is the precondition a mixture needs. SICK-R is
never trained on and is the honest transfer read.

## Why the dispatch works here

Dispatched amplitude is `(w_k/z) * sigmoid(gate_k) * consume_k(x)`, where
`w_k/z = sinh(u_k) / SUM_j cosh(u_j)` over ALL anchors (the damping law).
That gives ~1.0 only when one anchor engages and the other **abstains**
(`u ~ 0`); if both fire it collapses to ~0.5 and the mixture delivers HALF of
what either member does alone.

Measured here: mean `|w/z|` moved from **.310/.380 (blend)** before alignment to
**.645/.223 (specialize)** after 800 keys-only steps, with no starvation
strikes. That flip is why the mixture beats its best member rather than damping
itself below it.

## Load

```python
import amoe
h = amoe.attach(trunk, ["amoe/moe/equiv.anchor.pt", "amoe/moe/simplify.anchor.pt"],
                dispatch="amoe/moe/captionbert-v2-moe.dispatch.pt",
                binding=CaptionBertV2Binding(d=512))   # from modeling_captionbert.py
base = h.detach()        # bit-exact or raises
```

All anchors disabled reproduces the bare trunk **bit-exact** (asserted at build
time). Masking never renormalizes -- that is the damping law, not an oversight.

Anchor `.pt` key layout is `{block}.{param}`. (`blocks.{site}.{param}` in the
amoe README is the *safetensors* layout, a different serializer.)

## Training

Anchors: MNRL, in-batch + hard negatives where the source has them, 1,500 steps
at batch 256, pure Adam wd=0 (`amoe.laws.make_optimizer`), fp32/TF32 off.
1,500 not 4,000: the first solo run's STS-B **peaked at step 1,000** and then
fell .0178 while SICK-R kept climbing -- 56,825 distinct anchors behind 200,000
draws (space/draws .284). Dispatch: 800 steps, routing keys only (1,536 params),
anchors frozen, starvation safeguard armed.

See `metrics.json` for the full table and the routing telemetry.