alephllm — Nano-Beatrix training runs
The training ground for the nano line of Beatrix: byte-level crafts under ten million parameters, trained from birth to find out
what a craft that small can hold, how it holds it, and how fast it trains. The crafts, the trainer and the model code live in the
library github.com/AbstractEyes/alephllm (package geolip.alephllm); the bed that
drives them lives in github.com/AbstractEyes/geolip-nano-beatrix-trainer. This
repository holds what training produces. It is the small sibling of
alephllm-mini-beatrix-training, where the 237M and 257M
teachers, Virgil (the splat) and Matelda (the softmax control), were trained.
bed/manifest.json every run: arm, seed, steps, context, parameters, held-out bits, the fetch, minutes, the weights path
bed/reports/<run>.json the full record of one run: the training curve, the held-out bits by byte class and by position in the
window, the planted fetch at each distance, the expert dispatch census, the time cost
bed/reports/README.md the results index: one table per context, built from the records
bed/weights/<run>/ model.safetensors + config.json (loadable with the library's AlephLM from the config)
The bed
Every run trains one small craft: six blocks of width 256, eight heads, a bank of seven experts of width 256 in every block,
7.7 to 9.7 million parameters, a byte vocabulary of 256. The text is fineweb-edu (the sample-10BT configuration, documents of 400
to 24,000 bytes packed back to back). Into the training rows the bed plants records: a sentence such as The secret code is 48213. and, some hundreds or thousands of bytes later, the same sentence again, whose value the craft can only get right by
looking it up. The optimizer is plain Adam with a short warmup and a cosine schedule.
At the end of a run the bed reads, on held-out documents: the bits a byte (overall, by byte class, and by position in the window, in bands from the start of the document); the planted fetch at several distances (the teacher-forced bits of the value, the greedy first byte, the whole value copied); the expert dispatch census by byte class; and the time cost (minutes, tokens a second).
The arms are an attention form times a bank form:
| attention form | what it is |
|---|---|
| softmax, learned positions | ordinary attention with a learned position table |
| softmax, rotary | ordinary attention with rotary positions |
| hub | every block a splat hub: a scan over the row with a signed address of a few anchors a book, the position carried by a phase on the codes when rotary is on |
| dual hub | the hub with two address masses a block |
| alternating | hub blocks and softmax blocks taking turns, rotary on one half or on both |
| bank form | what it is |
|---|---|
| free | the experts chosen by the aleph address alone |
| fixed byte-class table | an installed table routes each byte class to its own expert; the learned dispatch is gone |
| both | the address and the table together |
What the first night showed (2026-10-12)
All runs below are from birth at a 4,096-byte window, 12,000 steps of 8 rows, five planted records a row, unless stated. The card was a shared desktop 4090, so the minutes are an upper bound; the tokens a second are the run's own count.
| form | expert table | parameters | held-out bits a byte | whole value copied at 64 / 512 / 1,024 / 2,048 / 3,072 bytes | minutes | tokens a second |
|---|---|---|---|---|---|---|
| softmax, learned positions | learned | 9.21M | 1.547 | 0.97 / 1.00 / 0.95 / 0.97 / 0.97 | 45.6 | 144k |
| softmax, rotary, seed 1 | learned | 9.21M | 1.535 | 0.97 / 0.86 / 0.80 / 0.97 / 0.98 | 19.2 | 341k |
| softmax, rotary, seed 2 | learned | 9.21M | 1.516 | 0.98 / 0.97 / 0.88 / 0.86 / 0.84 | 29.9 | 220k |
| softmax, rotary | fixed byte-class | 9.21M | 1.532 | 0.98 / 0.97 / 0.89 / 0.86 / 0.66 | 17.1 | 384k |
| alternating, rotary on the softmax half only | learned | 8.85M | 1.576 | 0.83 / 0.58 / 0.16 / 0.06 / 0.00 | 14.7 | - |
| alternating, rotary on both halves, seed 1 | learned | 8.85M | 1.567 | 0.97 / 0.98 / 0.94 / 0.98 / 0.94 | 13.6 | 481k |
| alternating, rotary on both halves, seed 2 | learned | 8.85M | 1.555 | 0.89 / 0.84 / 0.78 / 0.69 / 0.47 | 16.1 | 407k |
| alternating, rotary on both halves | fixed byte-class | 8.85M | 1.555 | 0.97 / 0.95 / 0.86 / 0.75 / 0.72 | 14.4 | 455k |
The softmax forms copy a planted value from three thousand bytes back from birth. The mixed craft does the same once the hub half carries the phase on its codes, at the fastest step of the set. The fixed byte-class table costs no text and no copy on either body and reads digits and capitals a few hundredths of a bit better.
The hub on its own needs pressure before it forms the lookup from birth at this window. The pressure is the number of positions each half-axis of the address must cover: 4,096 bytes divided by twice the anchors a book.
| hub form (rotary on the codes) | positions a half-axis | parameters | held-out bits | first byte at 64 / 512 / 1,024 / 2,048 / 3,072 | whole value | minutes |
|---|---|---|---|---|---|---|
| 64 anchors a book | 32 | 8.65M | 1.730 | 0.05 / 0.11 / 0.09 / 0.16 / 0.11 | 0 everywhere | 42.3 |
| 128 anchors | 16 | 8.67M | 1.689 | 0.05 / 0.08 / 0.09 / 0.14 / 0.11 | 0 everywhere | 21.7 |
| 256 anchors | 8 | 8.72M | 1.690 | 1.00 / 0.70 / 0.72 / 0.50 / 0.12 | 0.05 / 0 / 0 / 0 / 0 | 35.4 |
| 512 anchors | 4 | 8.82M | 1.658 | 0.95 / 0.95 / 0.98 / 1.00 / 0.97 | 0.59 / 0.66 / 0.56 / 0.66 / 0.52 | 52.0 |
| four books of 64 | 32 | 8.87M | 1.717 | 0.06 / 0.16 / 0.09 / 0.17 / 0.14 | 0 everywhere | 22.1 |
| eight books of 64 | 32 | 9.31M | 1.721 | 0.16 / 0.14 / 0.09 / 0.11 / 0.11 | 0 everywhere | 31.0 |
| 64 anchors, 24,000 steps in one schedule | 32 | 8.65M | 1.593 | 0.98 / 1.00 / 0.47 / 0.11 / 0.09 | 0.08 / 0.05 / 0 / 0 / 0 | 33.8 |
| 64 anchors, formed at a 1,024 window first, then 12,000 steps at 4,096 | 32 | 8.65M | 1.681 | 0.89 / 0.89 / 0.94 / 0.95 / 0.66 | 0.62 / 0.66 / 0.72 / 0.50 / 0.50 | 17.9 |
| two masses of 64, 12,000 steps and then 12,000 more | 32 | 9.66M | 1.572 | 1.00 everywhere | 0.97 / 0.91 / 0.89 / 0.91 / 0.94 | 24.9 + 30.3 |
Three things follow. There is a threshold between 16 and 8 positions a half-axis, not a slope: below it the width keeps paying, and at 4 the whole value forms at every distance. Books add parameters, not width: four or eight books of 64 form nothing where one book of 512 forms everything. Two roads open the narrow address anyway: form the lookup at a short window and extend, or give the block two masses. Whether the second mass or the fresh learning-rate schedule did the forming in the last row is one control away and is queued.
The byte-class table at a 1,024 window (2,500 steps; the back-to-back timing on a free card gave 58.1 against 65.5 ms a step and 0.46 against 0.55 ms an inference row):
| body | expert table | held-out bits | digit bits |
|---|---|---|---|
| softmax, learned positions | learned | 1.889 | 4.06 |
| softmax, learned positions | fixed | 1.892 | 3.99 |
| softmax, rotary | learned | 1.792 | 4.00 |
| softmax, rotary | fixed | 1.790 | 3.96 |
Bits by position in the window (held-out documents, bands in bytes from the document start):
| craft at 4,096 | 0-64 | 64-256 | 256-1,024 | 1,024-2,048 | 2,048-4,096 |
|---|---|---|---|---|---|
| softmax, rotary, seed 2 | 2.14 | 1.63 | 1.55 | 1.49 | 1.49 |
| alternating, rotary on both halves, fixed table | 2.15 | 1.67 | 1.57 | 1.53 | 1.54 |
| two masses of 64, 24,000 steps | 2.17 | 1.66 | 1.59 | 1.56 | 1.55 |
Ordinary text gets cheaper out to about a thousand or two thousand bytes and is flat past that at this size, while the planted lookup reaches three thousand bytes. The far half of a long window pays for memory, not for prose, in a craft this small.
What comes next
The frame beds: rows built from frames of generic sizes (32, 128, 256 bytes and up) opened and closed by the byte registry's own tags, nested and overlapping, with records to look up from the window or to recall from the weights; the capacity curves that say what the craft holds; the twin pair, Castor (the rotary softmax body) and Pollux (the all-rotary alternating body), both with the byte-class table beside the aleph address (the bank form two seeds settled), trained under the same recipe and read side by side. Castor is the mortal twin whose memory grows with the window; Pollux the immortal one whose hub half keeps a constant decode memory.
Licence and credit
The weights are CC BY-NC-SA 4.0 and the code MIT, as LICENSE.md states; the training text is fineweb-edu (ODC-By 1.0). Direction and design by AbstractPhil; the experiments, the bed and these records built in collaboration with Claude (Anthropic).