---
license: mit
language: [en]
pipeline_tag: feature-extraction
tags: [sentence-similarity, feature-extraction, consensus-distillation, mixture-of-experts, sparse-routing, geometric-deep-learning, amoe, research-log]
datasets: [AbstractPhil/conceptual-captions-12m-webdataset-berts, AbstractPhil/captionbert-8192-v2-consensus]
base_model: [google-bert/bert-base-uncased, answerdotai/ModernBERT-base, FacebookAI/roberta-base, albert/albert-base-v2, distilbert/distilbert-base-uncased]
---

# AlephLM-0 — an anchored expert trunk, distilled against a dense control

**This is a live experiment repository, not a finished model release.** Runs land
here as they finish training, checkpoints push every 30 minutes mid-run, and
*every* arm ships — including any that end up refuted. If you are reading this
while the run table below says IN PROGRESS, you are watching the experiment
happen.

## The question

Mixture-of-experts models normally route with a learned softmax over expert
logits — a *comparative* choice among experts. This program tests a different
router: a **closed-form signed address** over unit anchor directions,

```
u_k = cos(x, a_k) / τ          w_k = sinh(u_k) / Σ_j cosh(u_j)
```

where each expert's contribution is `w_k · σ(g_k) · E_k(x)` per **token**. The
weights are signed — an expert can be recruited *negatively* (an inhibitory
anchor) — and the read is reconstructive rather than competitive: no argmax, no
top-k, no load-balancing loss. The anchors, gates, and experts are trained by
nothing but the task gradient.

**E1 (this repo):** does a trunk built this way match or beat a
parameter-matched dense trunk under an identical objective, at 32M-row scale?
Six runs answer it:

| run | encoder | routing | seeds |
|---|---|---|---|
| `a1_anchored` | trunk-expert ff512 + 3 dispatched experts ff512/block | signed aleph address, learned anchors | s0, s1 |
| `a2_dense` | standard dense ff2048 | — (the control) | s0, s1 |
| `a3_random` | same as a1 | anchors **frozen at random init** | s0, s1 |

a1 vs a2 is the headline; a1 vs a3 isolates whether *learned* addressing
matters or any fixed partition of the capacity would do.

## Architecture

12 layers, d=512, 8 heads, pre-norm, 8192 learned positions, 768-d projected
output, CLS readout (settled empirically — see S0e below).

- Per block, the dense FFN (ff2048) is replaced by **1 always-on trunk expert
  (ff512) + 3 dispatched experts (ff512 each)** — 2048 hidden units total,
  exact capacity parity with the control.
- Dispatched-expert **output layers are zero-initialized** and gates start at
  σ(−3) ≈ 0.047: at initialization the dispatch contributes *exactly zero*
  (bit-exact, asserted at construction), so the anchored trunk is born as its
  own dense-trunk null hypothesis and the routing must earn its way in. One
  known consequence: the routing gradient is zero for exactly one step
  (∂L/∂w = σ(g)·E(x) and E ≡ 0 at init), the same dynamic as LoRA's A-matrix
  under B=0.
- Parameter cost of the machinery: **+36,900 over dense (+0.063%)** — 12
  codebooks of 3×512, 36 gates, and the extra expert biases.
  58,345,764 vs 58,308,864.

## Training recipe (identical for every arm)

Consensus distillation, inherited verbatim from
[captionbert-8192-v2](https://huggingface.co/AbstractPhil/captionbert-8192-v2):
the target for each caption is the L2-normalized centroid of five BERT-family
teachers, each mapped into the reference member's frame (bert-base) by a
whitened Procrustes fit — the
[precomputed targets](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus)
cover 33M captions from
[CC12M](https://huggingface.co/datasets/AbstractPhil/conceptual-captions-12m-webdataset-berts).

- loss = InfoNCE(T=0.07, in-batch negatives) + MSE (`F.mse_loss`, per-element
  mean — the batch of 2048 **is** the negative set, so batch size is part of
  the objective and is never changed)
- pure Adam (no weight decay), lr 6e-4, linear warmup 2000 → cosine to 1e-6,
  grad clip 1.0, AMP fp16, 4 epochs over 64 train chunks (31.9M rows), 2
  holdout chunks for eval
- length-bucketed dynamic padding (ceiling 256 tokens), gradient checkpointing
- trained on a single RTX 5090 (32GB); worst-case batch measured 30.1 GB
  reserved

## Stage-0 instruments (complete)

**S0a — is the rank ceiling the teachers' agreement, or bert's own geometry?**
(`s0a/s0a_erank.json`) The consensus target occupies an effective rank of
**28.1**/768. Raw bert-base rows on the same corpus: **40.7**/768 — and
**40.3** on out-of-domain STS-B text, so the low rank is the encoder's
geometry, not the corpus. Verdict at the matched (L2-normalized) gauge:
ratio 1.45× → *intermediate* — the consensus construction costs ~30% of the
member's rank, but the member itself only has ~40 directions to give. Any
consensus built in a bert frame is capped near 40 regardless of teacher
roster.

**S0e — pooling settle** (`runs/alephlm0-s0e-*`). Three identical dense
trunks, one seed shared exactly (same init, same batch plan), differing only
in readout, 500k rows × 2 epochs:

| readout | cos→target | mimicry R@1 |
|---|---|---|
| mean over mask | .6037 | .7745 |
| **CLS token** | **.6147** | **.8180** |
| learned-query attention | .6033 | .7680 |

CLS wins both gauges, outside the preregistered tie band (.003 cos / .01 R@1)
— notable because the *target* is a mean-pooled object, and the attention
readout (initialized to be exactly mean pooling) declined to move away from
mean. Stage 1 therefore trains with the CLS readout.

## Run status

| run | status |
|---|---|
| `runs/alephlm0-s0e-{mean,cls,attn}` | ✅ complete |
| `s0a/` erank instrument | ✅ complete |
| `runs/alephlm0-a2_dense-s0` | ✅ complete — mimicry R@1 .9975, cos→target .8418, erank 99.2/768; **8-task capability .6026** (`eval/`), inside the captionbert-v2/-B band: the dense control is triple-replicated |
| `runs/alephlm0-a3_random-s0` | ✅ complete — mimicry .9980, cos→target .8392, erank 98.8; **capability .6033** (band center: frozen-random routing matches dense at capacity parity); **dispatch-OFF .5772** — the routed experts carry −.026 of task function, degrading gracefully (`eval/`) |
| `runs/alephlm0-a1_anchored-s0` | ✅ complete — mimicry .9980, cos→target .8394, erank 98.6; **capability .6031**, dispatch-OFF .5743 (toggle −.0288). Anchors moved 1.06 rad from init; amplitude .101 |
| `runs/alephlm0-a1_anchored-s1` | 🔄 IN PROGRESS |
| `runs/alephlm0-{a2,a3}-s1` | queued |

**E1 at seed 0 (replication in flight): a three-way tie.** Learned-anchor,
frozen-anchor, and dense trunks land within .0007 of each other on the
8-task mean at exact capacity parity — the anchored form costs nothing,
and learned addressing does not exceed a frozen random partition *as an
encoder trunk*, even though it visibly reorganizes (anchors rotate a full
radian, and its dispatched experts carry more function than the frozen
arm's by the toggle gauge). The signed-address form's predicted advantage
lives where the address parameterizes the output distribution — that is
Stage 2's generative bed, which this result gates nothing about.

Each run directory carries `checkpoints/` (state + rolling model snapshots +
`final_model.pt` + `metrics.json`), `config/` (the exact resolved
configuration), and `tensorboard/`. Anchored runs additionally log per-block
**routing vitals** at every eval: mean dispatched amplitude |w·σ(g)|, anchor
drift from initialization, gate openings, and address-usage diversity — the
curves that show the routing waking from its zero-initialized silence.

## Lineage

- Teachers: bert-base-uncased, ModernBERT-base, roberta-base, albert-base-v2,
  distilbert-base-uncased (mean-pooled, 512-token truncation)
- Dense-recipe provenance: [captionbert-8192-v2](https://huggingface.co/AbstractPhil/captionbert-8192-v2)
  (.6077 8-task STS mean, beating its best teacher at 13% of the combined
  teacher parameters) and its replication
  [captionbert-8192-v2-B](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B)
- The signed-address form and its training laws come from a long-running
  research program on geometric routing (AMOE); the amplitude-conservation
  result that motivates per-token signed dispatch was established on adapter
  collectives before being carried inward here.

*Maintained as a live research log. Numbers in this card are measured, not
projected; anything not yet measured is marked as such.*