| --- |
| license: mit |
| language: [en] |
| tags: [aleph, mixture-of-experts, linear-attention, byte-level, research-log, tensorboard] |
| --- |
| |
| # alephllm β Mini-Beatrix training runs |
|
|
| Live training ground for **AlephLLM**: signed-address (aleph) language |
| models. Code, presets, trainer, and the full test array live in the |
| source repo β [github.com/AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm) |
| (`pip install git+https://github.com/AbstractEyes/alephllm`, package |
| `geolip.alephllm`). This repo holds what training produces, one prefix |
| per craft: |
|
|
| ``` |
| <craft>/manifest.json what is trained, what is planned, |
| tokens run, phase statuses, ckpt index |
| <craft>/checkpoints/step_XXXXXXXX.safetensors bf16 weights |
| <craft>/checkpoints/fp8/step_XXXXXXXX.safetensors fp8-e4m3 shipping variant |
| (inference testing only β never train from these) |
| <craft>/resume/latest.pt full resume state: fp32 model, Muon+Adam |
| optimizer states, data-stream position, RNG |
| <craft>/runs/ TensorBoard event files (rendered in the |
| Training metrics tab) |
| ``` |
|
|
| ## The mission ladder |
|
|
| | craft | d / L / ctx | params | tokenizer | role | |
| |---|---|---|---|---| |
| | mini-beatrix-0 | 512 / 12 / 1024 | 37.6M | byte-trigram | gate craft β its first toggle evals are the anchored-bank-under-AR screen | |
| | mini-beatrix-1 | 768 / 16 / 2048 | 112.5M | byte-trigram | first Colab mission | |
| | mini-beatrix-2 | 1024 / 20 / 2048 | 249.1M | byte-trigram | second mission | |
| | beatrix-voyager | 1536 / 24 / 4096 | 775.3M | BPE (gpt2) | flagship, gated on the mini verdicts | |
|
|
| Each craft has a `*-control` twin (identical minus the aleph attention |
| layers) β the running architecture control. Training is **resume-first**: |
| sessions are manually triggered on Colab (RTX 6000 Pro, 96GB, bf16), each |
| session pulls `manifest.json` + `resume/latest.pt` and continues where the |
| last one stopped. |
|
|
| ## The architecture in one paragraph |
|
|
| Trigram byte embedding (dedicated pad row) β pre-norm stack of standard |
| causal SDPA plus three **CausalSplatHUB** layers (causal *linear* |
| attention through a 2K-half-axis signed address, exact chunked scan) β |
| per-layer **anchored FFN bank** (always-on trunk + 3 dispatched experts, |
| expert outputs zero-init so the dispatch is born contributing exactly |
| zero, gates Ο(β3), no balance machinery) β dual head whose aleph read |
| enters at Ξ³=0 and must earn its way in by gradient. Muon on transport |
| weights + pure Adam elsewhere; flat LR; bf16 autocast over fp32 masters; |
| fp8 is a shipping format, never a training format. |
|
|
| ## Reading the instruments |
|
|
| TensorBoard carries the full born-in gauge suite: per-layer hidden-state |
| effective rank, consumed-address erank per hub layer, coefficient-of- |
| variation load analysis per bank, sign census, gate/Ξ³ trajectories, |
| anchor drift, denominator health, structural collapse flags (anchor |
| merging, dispatch-entropy collapse, erank floor, loss spikes), canary |
| recall (clean-protocol in-context binding probes), and the **toggle |
| ledger** β bpb deltas with each aleph mechanism switched off, the causal |
| record of what the addresses actually contribute. |
|
|
| Related record: the attention-side measurement campaign lives at |
| [aleph-splat-0](https://huggingface.co/AbstractPhil/aleph-splat-0); the |
| encoder-side anchored-bank record at |
| [alephlm-0](https://huggingface.co/AbstractPhil/alephlm-0). |
|
|
