File size: 3,181 Bytes
d40b627 b007aec d40b627 b007aec | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | ---
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- byte-level
- tokenizer-free
- aleph
- signed-address
- custom_code
---
# mini-beatrix-1 — pretrain annealment point (pre-classroom)
The locked pretrain+anneal state of **mini-beatrix-1**, a 112.5M-parameter
**byte-level** AlephLM: step **58,664**, **17.301B bytes** seen
(0.3B wikitext warmup · 15B fineweb-edu · 2B anneal mix), fineweb-holdout
val **1.045 bits/byte**. This checkpoint is the fixed departure point for
the staged "early-life curriculum" — later classroom checkpoints live in
the [training repo](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training).
No tokenizer: she reads raw UTF-8 bytes (`input_ids` = byte values
0–255). Each position composes a byte trigram (dedicated pad row), so
"tokens" are learned inside the network. Sixteen pre-norm layers where
routing uses **signed geometric addresses** — `sinh/Σcosh` dispatch over
learned unit anchors, inhibition as a first-class citizen, no softmax-
over-choices, no top-k, no balance losses. Each layer carries an
anchored FFN bank born contributing exactly zero; layers 4/9/14 use a
linear-cost address read (CausalSplatHUB) instead of softmax attention.
Both elected themselves into load-bearing work: at this checkpoint,
removing the banks costs **+2.25 bpb**, removing the hub attention
**+3.73 bpb** (toggle ledger, fineweb holdout). The dual head's aleph
read is present with its gate folded to 1.0 (a verified semantic no-op,
max|logit diff| 2.4e-07) and contributes 0.0000 bpb here — it is the
live subject of the head-election experiment in the classroom phase.
## Use
```python
import torch
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(
"AbstractPhil/mini-beatrix-1", trust_remote_code=True).eval()
ids = torch.tensor([list("The history of mathematics begins".encode())])
out = m.generate(ids, max_new_tokens=96, do_sample=True,
temperature=0.7, top_p=0.95)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))
```
Bits-per-byte on your own text: pass `labels=input_ids` (HF shift
semantics are internal) and divide the returned loss (nats/byte) by
`ln 2`. No KV cache in this wrapper — generation recomputes the prefix
each step; for cached decode use the native stack below.
## Honest notes
- The 2B anneal mix included dialogue in her chat template and a small
identity texture, so **the bare model chats and knows her name** —
behavior we have since ruled OUT of core corpora (conditioning belongs
in detachable arms; see the [amoe-lora](https://github.com/AbstractEyes/amoe-lora)
arm system and `mini-beatrix-1/arms/` in the training repo).
- Small and early: conversational in shape, thin on knowledge,
confidently wrong at times. Curriculum probe baselines (P0–P8),
toggle ledgers, and lexicon-census reports for this exact checkpoint
are in the training repo under `mini-beatrix-1/reports/`.
*Code:* [github.com/AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm) ·
*talk to her:* [alephllm-chat](https://huggingface.co/spaces/AbstractPhil/alephllm-chat)
|