license: mit
language:
- en
tags:
- aleph
- mixture-of-experts
- linear-attention
- byte-level
- research-log
- tensorboard
alephllm — Mini-Beatrix training runs
Live training ground for AlephLLM: signed-address (aleph) language
models. Code, presets, trainer, and the full test array live in the
source repo — github.com/AbstractEyes/alephllm
(pip install git+https://github.com/AbstractEyes/alephllm, package
geolip.alephllm). This repo holds what training produces, one prefix
per craft:
<craft>/manifest.json what is trained, what is planned,
tokens run, phase statuses, ckpt index
<craft>/checkpoints/step_XXXXXXXX.safetensors bf16 weights
<craft>/checkpoints/fp8/step_XXXXXXXX.safetensors fp8-e4m3 shipping variant
(inference testing only — never train from these)
<craft>/resume/latest.pt full resume state: fp32 model, Muon+Adam
optimizer states, data-stream position, RNG
<craft>/runs/ TensorBoard event files (rendered in the
Training metrics tab)
The mission ladder
| craft | d / L / ctx | params | tokenizer | role |
|---|---|---|---|---|
| mini-beatrix-0 | 512 / 12 / 1024 | 37.6M | byte-trigram | gate craft — its first toggle evals are the anchored-bank-under-AR screen |
| mini-beatrix-1 | 768 / 16 / 2048 | 112.5M | byte-trigram | first Colab mission |
| mini-beatrix-2 | 1024 / 20 / 2048 | 249.1M | byte-trigram | second mission |
| beatrix-voyager | 1536 / 24 / 4096 | 775.3M | BPE (gpt2) | flagship, gated on the mini verdicts |
Each craft has a *-control twin (identical minus the aleph attention
layers) — the running architecture control. Training is resume-first:
sessions are manually triggered on Colab (RTX 6000 Pro, 96GB, bf16), each
session pulls manifest.json + resume/latest.pt and continues where the
last one stopped.
The architecture in one paragraph
Trigram byte embedding (dedicated pad row) → pre-norm stack of standard causal SDPA plus three CausalSplatHUB layers (causal linear attention through a 2K-half-axis signed address, exact chunked scan) → per-layer anchored FFN bank (always-on trunk + 3 dispatched experts, expert outputs zero-init so the dispatch is born contributing exactly zero, gates σ(−3), no balance machinery) → dual head whose aleph read enters at γ=0 and must earn its way in by gradient. Muon on transport weights + pure Adam elsewhere; flat LR; bf16 autocast over fp32 masters; fp8 is a shipping format, never a training format.
Reading the instruments
TensorBoard carries the full born-in gauge suite: per-layer hidden-state effective rank, consumed-address erank per hub layer, coefficient-of- variation load analysis per bank, sign census, gate/γ trajectories, anchor drift, denominator health, structural collapse flags (anchor merging, dispatch-entropy collapse, erank floor, loss spikes), canary recall (clean-protocol in-context binding probes), and the toggle ledger — bpb deltas with each aleph mechanism switched off, the causal record of what the addresses actually contribute.
Related record: the attention-side measurement campaign lives at aleph-splat-0; the encoder-side anchored-bank record at alephlm-0.