---
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
  - byte-level
  - tokenizer-free
  - aleph
  - signed-address
  - custom_code
---

# mini-beatrix-1 — pretrain annealment point (pre-classroom)

The locked pretrain+anneal state of **mini-beatrix-1**, a 112.5M-parameter
**byte-level** AlephLM: step **58,664**, **17.301B bytes** seen
(0.3B wikitext warmup · 15B fineweb-edu · 2B anneal mix), fineweb-holdout
val **1.045 bits/byte**. This checkpoint is the fixed departure point for
the staged "early-life curriculum" — later classroom checkpoints live in
the [training repo](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training).

No tokenizer: she reads raw UTF-8 bytes (`input_ids` = byte values
0–255). Each position composes a byte trigram (dedicated pad row), so
"tokens" are learned inside the network. Sixteen pre-norm layers where
routing uses **signed geometric addresses** — `sinh/Σcosh` dispatch over
learned unit anchors, inhibition as a first-class citizen, no softmax-
over-choices, no top-k, no balance losses. Each layer carries an
anchored FFN bank born contributing exactly zero; layers 4/9/14 use a
linear-cost address read (CausalSplatHUB) instead of softmax attention.
Both elected themselves into load-bearing work: at this checkpoint,
removing the banks costs **+2.25 bpb**, removing the hub attention
**+3.73 bpb** (toggle ledger, fineweb holdout). The dual head's aleph
read is present with its gate folded to 1.0 (a verified semantic no-op,
max|logit diff| 2.4e-07) and contributes 0.0000 bpb here — it is the
live subject of the head-election experiment in the classroom phase.

## Use

```python
import torch
from transformers import AutoModelForCausalLM

m = AutoModelForCausalLM.from_pretrained(
    "AbstractPhil/mini-beatrix-1", trust_remote_code=True).eval()

ids = torch.tensor([list("The history of mathematics begins".encode())])
out = m.generate(ids, max_new_tokens=96, do_sample=True,
                 temperature=0.7, top_p=0.95)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))
```

Bits-per-byte on your own text: pass `labels=input_ids` (HF shift
semantics are internal) and divide the returned loss (nats/byte) by
`ln 2`. No KV cache in this wrapper — generation recomputes the prefix
each step; for cached decode use the native stack below.

## Honest notes

- The 2B anneal mix included dialogue in her chat template and a small
  identity texture, so **the bare model chats and knows her name** —
  behavior we have since ruled OUT of core corpora (conditioning belongs
  in detachable arms; see the [amoe-lora](https://github.com/AbstractEyes/amoe-lora)
  arm system and `mini-beatrix-1/arms/` in the training repo).
- Small and early: conversational in shape, thin on knowledge,
  confidently wrong at times. Curriculum probe baselines (P0–P8),
  toggle ledgers, and lexicon-census reports for this exact checkpoint
  are in the training repo under `mini-beatrix-1/reports/`.

*Code:* [github.com/AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm) ·
*talk to her:* [alephllm-chat](https://huggingface.co/spaces/AbstractPhil/alephllm-chat)