File size: 3,561 Bytes
e9a395d a443286 e9a395d a443286 e657c40 a443286 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: mit
language: [en]
tags: [aleph, mixture-of-experts, linear-attention, byte-level, research-log, tensorboard]
---
# alephllm — Mini-Beatrix training runs
Live training ground for **AlephLLM**: signed-address (aleph) language
models. Code, presets, trainer, and the full test array live in the
source repo — [github.com/AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm)
(`pip install git+https://github.com/AbstractEyes/alephllm`, package
`geolip.alephllm`). This repo holds what training produces, one prefix
per craft:
```
<craft>/manifest.json what is trained, what is planned,
tokens run, phase statuses, ckpt index
<craft>/checkpoints/step_XXXXXXXX.safetensors bf16 weights
<craft>/checkpoints/fp8/step_XXXXXXXX.safetensors fp8-e4m3 shipping variant
(inference testing only — never train from these)
<craft>/resume/latest.pt full resume state: fp32 model, Muon+Adam
optimizer states, data-stream position, RNG
<craft>/runs/ TensorBoard event files (rendered in the
Training metrics tab)
```
## The mission ladder
| craft | d / L / ctx | params | tokenizer | role |
|---|---|---|---|---|
| mini-beatrix-0 | 512 / 12 / 1024 | 37.6M | byte-trigram | gate craft — its first toggle evals are the anchored-bank-under-AR screen |
| mini-beatrix-1 | 768 / 16 / 2048 | 112.5M | byte-trigram | first Colab mission |
| mini-beatrix-2 | 1024 / 20 / 2048 | 249.1M | byte-trigram | second mission |
| beatrix-voyager | 1536 / 24 / 4096 | 775.3M | BPE (gpt2) | flagship, gated on the mini verdicts |
Each craft has a `*-control` twin (identical minus the aleph attention
layers) — the running architecture control. Training is **resume-first**:
sessions are manually triggered on Colab (RTX 6000 Pro, 96GB, bf16), each
session pulls `manifest.json` + `resume/latest.pt` and continues where the
last one stopped.
## The architecture in one paragraph
Trigram byte embedding (dedicated pad row) → pre-norm stack of standard
causal SDPA plus three **CausalSplatHUB** layers (causal *linear*
attention through a 2K-half-axis signed address, exact chunked scan) →
per-layer **anchored FFN bank** (always-on trunk + 3 dispatched experts,
expert outputs zero-init so the dispatch is born contributing exactly
zero, gates σ(−3), no balance machinery) → dual head whose aleph read
enters at γ=0 and must earn its way in by gradient. Muon on transport
weights + pure Adam elsewhere; flat LR; bf16 autocast over fp32 masters;
fp8 is a shipping format, never a training format.
## Reading the instruments
TensorBoard carries the full born-in gauge suite: per-layer hidden-state
effective rank, consumed-address erank per hub layer, coefficient-of-
variation load analysis per bank, sign census, gate/γ trajectories,
anchor drift, denominator health, structural collapse flags (anchor
merging, dispatch-entropy collapse, erank floor, loss spikes), canary
recall (clean-protocol in-context binding probes), and the **toggle
ledger** — bpb deltas with each aleph mechanism switched off, the causal
record of what the addresses actually contribute.
Related record: the attention-side measurement campaign lives at
[aleph-splat-0](https://huggingface.co/AbstractPhil/aleph-splat-0); the
encoder-side anchored-bank record at
[alephlm-0](https://huggingface.co/AbstractPhil/alephlm-0).
|