---
license: mit
language: [en]
tags: [aleph, mixture-of-experts, linear-attention, byte-level, research-log, tensorboard]
---

# alephllm — Mini-Beatrix training runs

Live training ground for **AlephLLM**: signed-address (aleph) language
models. Code, presets, trainer, and the full test array live in the
source repo — [github.com/AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm)
(`pip install git+https://github.com/AbstractEyes/alephllm`, package
`geolip.alephllm`). This repo holds what training produces, one prefix
per craft:

```
<craft>/manifest.json                      what is trained, what is planned,
                                           tokens run, phase statuses, ckpt index
<craft>/checkpoints/step_XXXXXXXX.safetensors    bf16 weights
<craft>/checkpoints/fp8/step_XXXXXXXX.safetensors  fp8-e4m3 shipping variant
                                           (inference testing only — never train from these)
<craft>/resume/latest.pt                   full resume state: fp32 model, Muon+Adam
                                           optimizer states, data-stream position, RNG
<craft>/runs/                              TensorBoard event files (rendered in the
                                           Training metrics tab)
```

## The mission ladder

| craft | d / L / ctx | params | tokenizer | role |
|---|---|---|---|---|
| mini-beatrix-0 | 512 / 12 / 1024 | 37.6M | byte-trigram | gate craft — its first toggle evals are the anchored-bank-under-AR screen |
| mini-beatrix-1 | 768 / 16 / 2048 | 112.5M | byte-trigram | first Colab mission |
| mini-beatrix-2 | 1024 / 20 / 2048 | 249.1M | byte-trigram | second mission |
| beatrix-voyager | 1536 / 24 / 4096 | 775.3M | BPE (gpt2) | flagship, gated on the mini verdicts |

Each craft has a `*-control` twin (identical minus the aleph attention
layers) — the running architecture control. Training is **resume-first**:
sessions are manually triggered on Colab (RTX 6000 Pro, 96GB, bf16), each
session pulls `manifest.json` + `resume/latest.pt` and continues where the
last one stopped.

## The architecture in one paragraph

Trigram byte embedding (dedicated pad row) → pre-norm stack of standard
causal SDPA plus three **CausalSplatHUB** layers (causal *linear*
attention through a 2K-half-axis signed address, exact chunked scan) →
per-layer **anchored FFN bank** (always-on trunk + 3 dispatched experts,
expert outputs zero-init so the dispatch is born contributing exactly
zero, gates σ(−3), no balance machinery) → dual head whose aleph read
enters at γ=0 and must earn its way in by gradient. Muon on transport
weights + pure Adam elsewhere; flat LR; bf16 autocast over fp32 masters;
fp8 is a shipping format, never a training format.

## Reading the instruments

TensorBoard carries the full born-in gauge suite: per-layer hidden-state
effective rank, consumed-address erank per hub layer, coefficient-of-
variation load analysis per bank, sign census, gate/γ trajectories,
anchor drift, denominator health, structural collapse flags (anchor
merging, dispatch-entropy collapse, erank floor, loss spikes), canary
recall (clean-protocol in-context binding probes), and the **toggle
ledger** — bpb deltas with each aleph mechanism switched off, the causal
record of what the addresses actually contribute.

Related record: the attention-side measurement campaign lives at
[aleph-splat-0](https://huggingface.co/AbstractPhil/aleph-splat-0); the
encoder-side anchored-bank record at
[alephlm-0](https://huggingface.co/AbstractPhil/alephlm-0).