README.md

---
license: mit
language: [en]
pipeline_tag: text-generation
tags: [adapters, knowledge-distillation, mixture-of-experts, geometric-deep-learning, amoe, research-log, qwen2.5]
base_model: [Qwen/Qwen2.5-0.5B-Instruct]
---

# alephlm-adopt-0 — an AlephLM by adoption

**A live experiment repository.** The question: can the aleph — a
closed-form *signed* addressing mechanism — carry the capability delta
between a small frozen LLM and its larger sibling, as a fully detachable
attachment? Here the trunk is **Qwen2.5-0.5B-Instruct, frozen entirely**;
the trained object is ~2.9M parameters of attachments distilled by
per-token KL from **Qwen2.5-1.5B-Instruct**.

Two param-matched arms, reported as a *contribution ledger* (no winner
declared — the arms turned out to be different mechanisms, not ranked
ones):

| arm | attachment | trainable |
|---|---|---|
| `aleph` | one 16-slot aleph relay per decoder block + one on the pre-head stream (the output-distribution placement) | 2,873,625 |
| `mlp` | param-matched plain adapters at the identical placements, zero-init outputs | 2,891,225 |

Each relay is a gated residual write, born nearly silent (gate σ(−3) ≈
0.047). The aleph relay reads its input against a codebook of unit
anchors with a **signed closed-form coefficient** (`sinh/Σcosh`) — an
anchor can contribute negatively — and writes back a reconstructive
read of the codebook; the mlp adapter is the standard
bottleneck-residual control.

## Distillation verdict (20k steps, ~31M tokens each, wikitext stream)

| config | ppl | KL→teacher | toggle off |
|---|---|---|---|
| frozen 0.5B (null) | 17.351 | 1.928 | — |
| + aleph | 15.449 | 1.448 | **recovers stock to 4 decimals** |
| + mlp | 15.394 | 1.432 | recovers stock to 4 decimals |

Both arms cut KL-to-teacher ~25% with ~0.6% of the trunk's parameter
count, and both are perfectly detachable — the cleanest toggle-law
result in this program at LLM scale.

**The mechanistic finding** (why no winner is declared): at equal
capability the two arms allocated *oppositely*. The aleph **opened its
pre-head relay 8×** (gate σ .047 → .387, its codebook rotating 0.47 rad)
— it elected the output-distribution placement as its main pathway. The
mlp **closed the same placement 10×** (σ → .004) and did its work in the
early blocks. One mechanism gravitates to where its lineage predicts its
advantage lives; the other flees it.

**Sign census** (the readout that motivated this repo's name): the
aleph relays run a *balanced bipolar code* — ~50% of coefficients
negative, |w| ≈ 1/K, dense across all 64 slots — a reconstructive sign
code, categorically different from softmax routing, and invisible to any
gauge that takes an absolute value.

## Standardized bench

0-shot via lm-eval-harness (acc_norm where defined, else acc):

| config | arc_easy | hellaswag | piqa | winogrande | mean |
|---|---|---|---|---|---|
| stock 0.5B | 0.5888 | 0.5251 | 0.7051 | 0.5564 | **0.5939** |
| + aleph | 0.5960 | 0.4669 | 0.6779 | 0.5675 | **0.5771** |
| + mlp | 0.5968 | 0.4743 | 0.6746 | 0.5462 | **0.5730** |
| teacher 1.5B | 0.7584 | 0.6822 | 0.7601 | 0.6290 | **0.7074** |

**The honest verdict**: the wikitext-KL distillation did not convert to
task capability — both arms trade ~.02 of 0-shot accuracy for their
~25% KL improvement, *identically*, closing none of the 0.5B→1.5B gap
(.11). The finding indicts the distillation **corpus/objective**, not
either mechanism: matching the teacher's distribution on encyclopedic
text is not the same thing as acquiring its task behavior. The
mechanistic differences between the arms (placement election, signed
regime, both fully detachable) survive as the real result of this
round; an instruct-domain distillation corpus is the named next
revision.

## Artifacts

- `adopt0/adopt0_aleph_s0.pt` — aleph relays + head relay (state dict
  keyed `block{i}.*` / `head.*`), with config and final metrics inside
- `adopt0/adopt0_mlp_s0.pt` — the param-matched control, same layout
- `adopt0/adopt0_results.json` — the distillation verdict table
- `adopt0/adopt0_bench.json` — the standardized bench (all four rows)

**Use**: load `Qwen/Qwen2.5-0.5B-Instruct`, wrap each decoder layer
with its `block{i}` relay as a residual on the block's hidden-state
output, wrap the final norm's output with the `head` relay, or load
nothing to recover stock behavior exactly. Training code ships in a
future revision.

## Lineage

The relay construction and its signed addressing come from a
long-running research program on geometric routing (AMOE); the
placement law motivating the pre-head relay — the signed address's
advantage concentrates where it parameterizes the output distribution —
was established on from-scratch LM substrates before this adoption
test. Sibling experiment: [alephlm-0](https://huggingface.co/AbstractPhil/alephlm-0),
the same addressing family trained *as* the trunk rather than bolted
onto one.

*Maintained as a live research log. Numbers are measured, not
projected; anything not yet measured is marked as such.*