AbstractPhil/alephllm-nano-castor-pollux

AlephLLM nano: Castor and Pollux, v1.1

Four small byte-level AlephLLM models (about 9 million parameters each, a 4,096-byte window), trained from birth on Beatrix 2s's diet: 16.1 billion bytes in thirteen phases (wikitext-103, fineweb-edu, nine curriculum stages, two anneals), 61,422 steps each. They read and write raw UTF-8 bytes: no tokenizer.

This is v1.1: the same four crafts from the same starts on the same diet, with the output layer's two readout matrices trained by AdamW (rate 0.0003, decoupled weight decay 0.1) instead of Muon. In the v1 run the readout kept growing under Muon and the last block's output gathered onto a single direction within its first thousand steps; that run was stopped, and its folders are archived in alephllm-nano-beatrix-training under archive/castor-pollux-v1/.

Beside each body sit its continuation arms (below): the same body taken further on a mix that teaches it to write and read its own notes. Every folder is a model with its full training record, and every folder loads the same way.

model	the body	parameters	bits a byte (fineweb-edu holdout)	recall: copy / nested / repeated key	1,024-byte frame	planted record at 2,048 / 3,584 bytes (bits)
nano-pollux-dual-canon-v1.1	Pollux's layout (hubs in blocks 0, 2, 4; rotary softmax in 1, 3, 5) with dual hubs and Canon layers	9,279,569	1.2927	0.33 / 0.33 / 0.12	0.03	45.59 / 49.67
nano-pollux-canon-v1.1	Pollux (rotary hubs and rotary softmax alternating) with Canon layers	8,860,241	1.2920	0.35 / 0.29 / 0.09	0.00	44.44 / 49.25
nano-pollux-dual-v1.1	Pollux's layout with dual hubs	9,267,281	1.3057	0.06 / 0.07 / 0.03	0.00	45.98 / 49.27
nano-castor-v1.1	Castor: rotary softmax in every block (the baseline)	9,215,057	1.2817	0.14 / 0.10 / 0.08	0.00	45.84 / 49.21

The recall columns are the nano bed's checks at the end of the train: a keyed record asked back whole (copy, a nested question, a key that was redefined), the same read on 1,024-byte frames, and the bits needed to recall a planted record from 2,048 and 3,584 bytes away (lower is better).

The continuation arms

Each body's diet ended at step 61,422. From there an arm takes the same body 3,815 steps further (1.0 billion bytes; the mixed arm 5,723 steps, 1.5 billion, the same optimizer settings) on a mix that teaches it to write and read its own notes; a second-stage arm continues an arm's end the same way (its folder suffix chains the two: -fr-cf is frames, then carry). Every arm folder is a model like the four above, with its own training record, and loads the same way.

arm	folder suffix	what it trains on
mixed	-mx	carry rows and framed rows side by side, their records in all three registers (letters, digits, hex), the plain text kept: one phase for the near read, the far habit and every register

One table a body: its folders, the steps each trained, and the nano bed's checks at each end on the same 192 held-out rows. The bed's held-out bits a byte is the bed's own fineweb-edu read (not the column of the first table). Far question, own carries: a keyed record early in the window and the question 2,048 bytes or more after it, the body free to re-write the record on the way; the share of rows answered whole, and carries a row how many re-writes it made. A dash: the read does not apply to that folder.

Pollux's layout (hubs in blocks 0, 2, 4; rotary softmax in 1, 3, 5) with dual hubs and Canon layers

folder	trained	the bed's held-out bits a byte	copy / nested / repeated key	1,024-byte frame	far question, own carries	carries a row
nano-pollux-dual-canon-v1.1	0 to 61,422 (the diet)	1.2717	0.33 / 0.33 / 0.12	0.03	-	-
nano-pollux-dual-canon-v1.1-mx	mixed: 61,422 to 67,145	1.2413	0.93 / 0.85 / 0.79	0.97	0.51	4.7

Pollux (rotary hubs and rotary softmax alternating) with Canon layers

folder	trained	the bed's held-out bits a byte	copy / nested / repeated key	1,024-byte frame	far question, own carries	carries a row
nano-pollux-canon-v1.1	0 to 61,422 (the diet)	1.2723	0.35 / 0.29 / 0.09	0.00	-	-
nano-pollux-canon-v1.1-mx	mixed: 61,422 to 67,145	1.2415	0.86 / 0.82 / 0.64	0.68	0.38	3.1

Pollux's layout with dual hubs

folder	trained	the bed's held-out bits a byte	copy / nested / repeated key	1,024-byte frame	far question, own carries	carries a row
nano-pollux-dual-v1.1	0 to 61,422 (the diet)	1.2861	0.06 / 0.07 / 0.03	0.00	-	-
nano-pollux-dual-v1.1-mx	mixed: 61,422 to 67,145	1.2537	0.81 / 0.77 / 0.66	0.65	0.57	4.4

Castor: rotary softmax in every block (the baseline)

folder	trained	the bed's held-out bits a byte	copy / nested / repeated key	1,024-byte frame	far question, own carries	carries a row
nano-castor-v1.1	0 to 61,422 (the diet)	1.2614	0.14 / 0.10 / 0.08	0.00	-	-
nano-castor-v1.1-mx	mixed: 61,422 to 67,145	1.2312	0.82 / 0.77 / 0.66	0.74	0.20	1.9

Training now: v1.2

This is v1.2: Pollux dual from the same start and with the same recipe as v1.1, on the same 16.1 billion bytes with half of every phase's rows framed from the first step (carry rows, system frames, caption windows, note rows): the frames are part of the diet, not a later phase. The plain v1.1 craft is its control. Castor's framed twin (nano-castor-v1.2, from nano-castor-v1's start, nano-castor-v1.1 its control) runs the same diet.

craft	the body	started from the start of its bed run
nano-castor-v1.2	Castor: rotary softmax in every block (the baseline)	castor-both, seed 3

Each folder fills as it trains (checkpoints/, resume/, manifest.json, runs/, bed_gauges/, and v1_boundaries.json with the bed's reads at every phase boundary) and becomes an AutoModel package like the folders above when it finishes. The trainer notebook: colab/nano_v1_2.ipynb.

Use

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("AbstractPhil/alephllm-nano-castor-pollux", subfolder="nano-pollux-dual-canon-v1.1",
                                             trust_remote_code=True)   # variant="fp32" for the fp32 weights
ids = torch.tensor([list("The history of astronomy".encode("utf-8"))])
out = model.generate(ids, max_new_tokens=64, do_sample=False)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

The model code sits at the top of the repo (transformers loads remote code from there) and again in each model's folder, so a folder downloaded alone loads with from_pretrained(<folder>, trust_remote_code=True). Each folder holds model.safetensors (bf16) and model.fp32.safetensors (the fp32 weights the training ended with); the bf16 file is the fp32 file rounded. Generation re-reads the whole window each step (no cache). config.json says what the model is (craft: its body, its arm, what it continued from, the phase and step its weights ended on; activated: the mechanisms its body switches on), and crafts.json at the top of the repo indexes every packaged folder with its end reads: the tables above are rendered from it.

Training

The recipe is Beatrix 2s's: Muon at 0.02 on the trunk's matrices, Adam at 0.0003 on the rest, 200 warmup steps then a flat rate, gradient clip 1.0, 262,144 bytes a step. The output layer's two readout matrices trained with AdamW (0.0003, decoupled weight decay 0.1) instead of Muon. Each model started from exactly the weights its strongest run on the nano bed started from. Each folder keeps the whole training record: checkpoints/, resume/, manifest.json, runs/ (TensorBoard: the training curves, the health census, the evals with bits by position, and the bed's checks at intervals under bed/), bed_gauges/ and v1_boundaries.json (one row a phase). An arm folder keeps the same record for its own steps: checkpoints at its two mid gauges and its end (an fp8 copy of the end), the end's resume state, the gauge records of its start state and its reads, manifest.json, runs/, v1_boundaries.json. The trainer: geolip-nano-beatrix-trainer (nano_v1.py, automodel.py, colab/nano_v1_1.ipynb, the arm notebooks colab/nano_v1_1_frames.ipynb, nano_v1_1_carry.ipynb, nano_v1_1_frames_carry.ipynb, nano_v1_1_notes.ipynb, and colab/nano_v1_1_release.ipynb, the packaging) on AlephLLM. The license follows the training data's share-alike and non-commercial terms, as Beatrix 2s's does.