AbstractPhil/alephllm-mini-beatrix-training

alephllm — Mini-Beatrix training runs

Live training ground for AlephLLM: signed-address (aleph) language models. Code, presets, trainer, and the full test array live in the source repo — github.com/AbstractEyes/alephllm (pip install git+https://github.com/AbstractEyes/alephllm, package geolip.alephllm). This repo holds what training produces, one prefix per craft:

<craft>/manifest.json                      what is trained, what is planned,
                                           tokens run, phase statuses, ckpt index
<craft>/checkpoints/step_XXXXXXXX.safetensors    bf16 weights
<craft>/checkpoints/fp8/step_XXXXXXXX.safetensors  fp8-e4m3 shipping variant
                                           (inference testing only — never train from these)
<craft>/resume/latest.pt                   full resume state: fp32 model, Muon+Adam
                                           optimizer states, data-stream position, RNG
<craft>/runs/                              TensorBoard event files (rendered in the
                                           Training metrics tab)

The mission ladder

craft	d / L / ctx	params	tokenizer	role
mini-beatrix-0	512 / 12 / 1024	37.6M	byte-trigram	gate craft — its first toggle evals are the anchored-bank-under-AR screen
mini-beatrix-1	768 / 16 / 2048	112.5M	byte-trigram	first Colab mission
mini-beatrix-2	1024 / 20 / 2048	249.1M	byte-trigram	second mission
beatrix-voyager	1536 / 24 / 4096	775.3M	BPE (gpt2)	flagship, gated on the mini verdicts

Each craft has a *-control twin (identical minus the aleph attention layers) — the running architecture control. Training is resume-first: sessions are manually triggered on Colab (RTX 6000 Pro, 96GB, bf16), each session pulls manifest.json + resume/latest.pt and continues where the last one stopped.

The architecture in one paragraph

Trigram byte embedding (dedicated pad row) → pre-norm stack of standard causal SDPA plus three CausalSplatHUB layers (causal linear attention through a 2K-half-axis signed address, exact chunked scan) → per-layer anchored FFN bank (always-on trunk + 3 dispatched experts, expert outputs zero-init so the dispatch is born contributing exactly zero, gates σ(−3), no balance machinery) → dual head whose aleph read enters at γ=0 and must earn its way in by gradient. Muon on transport weights + pure Adam elsewhere; flat LR; bf16 autocast over fp32 masters; fp8 is a shipping format, never a training format.

Reading the instruments

TensorBoard carries the full born-in gauge suite: per-layer hidden-state effective rank, consumed-address erank per hub layer, coefficient-of- variation load analysis per bank, sign census, gate/γ trajectories, anchor drift, denominator health, structural collapse flags (anchor merging, dispatch-entropy collapse, erank floor, loss spikes), canary recall (clean-protocol in-context binding probes), and the toggle ledger — bpb deltas with each aleph mechanism switched off, the causal record of what the addresses actually contribute.

Related record: the attention-side measurement campaign lives at aleph-splat-0; the encoder-side anchored-bank record at alephlm-0.