File size: 9,767 Bytes
86723d6
 
a9cf22d
 
 
629823e
 
 
 
 
a9cf22d
 
 
 
86723d6
629823e
 
 
88495b5
629823e
2315c01
 
 
a9cf22d
 
629823e
 
 
 
 
88495b5
629823e
a9cf22d
629823e
36ce8b6
a9cf22d
 
36ce8b6
 
a9cf22d
 
 
 
 
629823e
a9cf22d
629823e
a9cf22d
 
 
 
 
 
 
2315c01
 
a9cf22d
 
2315c01
 
 
ce2a9da
 
2315c01
 
ce2a9da
 
2315c01
 
a9cf22d
 
 
2315c01
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ce2a9da
 
2315c01
 
 
 
 
 
ce2a9da
2315c01
 
 
 
ce2a9da
 
 
 
 
 
d06baff
 
 
 
 
 
 
 
 
 
 
 
 
 
2315c01
b494981
 
2315c01
 
b494981
 
 
 
 
 
 
 
 
2315c01
 
 
a9cf22d
 
2315c01
d06baff
2315c01
a9cf22d
 
 
 
 
 
 
 
 
 
629823e
 
 
88495b5
629823e
 
 
a9cf22d
 
 
629823e
 
 
a9cf22d
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
---
license: apache-2.0
language:
- en
- ja
tags:
- exealm
- open-source
- from-scratch
- research-preview
- tonn
- hybrid
- retrieval
- multi-agent
---

# ExeaLM

An open language model built by high schoolers, run by [Exea Labs](https://exealabs.org).

**Status: research preview.** The planned 0.1B model is not trained yet. This repo holds the tokenizer, the
research checkpoints of the 17M-parameter experiments (`checkpoints/17m/`, below) and every result file
(`results/`). Everything is reproducible from
[github.com/exea-labs/ExeaLM](https://github.com/exea-labs/ExeaLM); the full report is
[results/README.md](https://github.com/exea-labs/ExeaLM/blob/main/results/README.md).

## Links

- Code: https://github.com/exea-labs/ExeaLM
- Contributor sign-up: https://docs.google.com/forms/d/e/1FAIpQLSdPHVZhuXxBDUf-cJM2zDbj1JZPsduu95Y_kTJ_g4gKAXdDVw/viewform
- Architecture paper: [TONN: Networks That Build Their Own Architecture from Data](https://www.preprints.org/manuscript/202611.0994)

## Model (planned)

- **Size:** 0.1 billion parameters (105M with the planned vocabulary, about 94M with this 32k tokenizer).
- **Architecture:** a hybrid of TONN and attention. Every third layer is a TONN layer: small learned circuits
  (convolutions, running sums, state spaces, memories) whose layout TONN's search adjusts during training. The other
  layers are multi-head attention with two parts from Qwen3.5: gated attention and QK-norm. Each layer is followed by
  a gated MLP.
- **Built-in memory circuits** in every TONN layer:
  - retrieval from a read-only library, with the model's own embeddings;
  - a read-write message board shared by several copies of the model, so agents can talk.
- **Training data:** about 100B tokens; see [DATA.md](https://github.com/exea-labs/ExeaLM/blob/main/DATA.md).
- **License:** Apache 2.0 for code and weights.

## Results so far (small-scale, on one Mac Studio GPU)

Bits per byte on held-out English, Japanese and code (lower is better).

| Size | Comparison | ExeaLM hybrid | Same-size Llama-style transformer |
|---|---|---|---|
| 13M | same training steps, 2 seeds | **1.531** | 1.554 |
| 27M | same training steps, best learning rates, 2 seeds | **1.492** | 1.523 |
| 48M | same training time, 2 seeds | **1.445** | 1.452 |
| ~0.1B (90M) | same training time, 2 seeds | **1.455** | 1.479 |
| 13M, with gated attention and QK-norm (from Qwen3.5) | same training time, 1 seed | **1.509** | 1.542 (1.515 with the same additions) |

- At 0.1B the hybrid trains at the same speed as the transformer (1.01 times its cost per token).
- One 18M model trained mostly on real text also passes facts between agents through its board (100% correct, 1%
  with the board cut) and looks facts up in a library it has never seen (94%, 1.6% without it), on both seeds. These
  are synthetic tasks.
- Gated attention and QK-norm also help the 18M model with both skills: 1.393 bits per byte averaged over the four
  held-out sets, against 1.419 without them (equal steps, one seed). It keeps the relay (100%) and lookup (93%).
- On real text the memory circuits do not pay off yet. On held-out Wikipedia, putting the right article in the
  context gains about 1 bit per re-mentioned rare word; the model's library and board gain between -0.1 and +0.06.
  The board path is also fragile: after 8,000 steps of practice on single articles, notes from all 256 test articles
  at once make re-mentions 7.9 bits worse.
- **Correction:** we reported that training as a swarm beats a matched control by 0.014 bits per byte. Reruns range
  from 0.015 better to 0.035 worse, so the swarm gain on real text is unproven.

All of these are short runs (about 50M training tokens), far smaller than the planned 100B-token run.

## Research checkpoints (`checkpoints/17m/`)

Small models from the experiments in
[results/README.md](https://github.com/exea-labs/ExeaLM/blob/main/results/README.md), sections 10 to 12. Each was trained
for 24,000 steps of 32 × 512 tokens (393M tokens) on one Mac Studio. **They are research artifacts, not a release:**
small, briefly trained, and not evaluated on any benchmark.

The ExeaLM checkpoints are the `small` hybrid at width 320: 6 layers, TONN layers at 0 and 3, each with a retrieval
circuit and an agent-board circuit (18.08M parameters). The Llama checkpoints are a same-size Llama-style transformer
(17.83M). Scores: English bits per byte (lower is better); relay through the board and lookup in an unseen library
(chance is 1.6%).

| File | Model and training | English | Relay | Lookup |
|---|---|---|---|---|
| `exealm_together_seed0.pt` | ExeaLM, both skills taught together from step 0 (the recipe to use) | 1.320 | 100.0% | 94.0% |
| `exealm_together_seed1.pt` | the same, seed 1 | 1.328 | 100.0% | 94.3% |
| `exealm_relayfirst_seed1.pt` | ExeaLM, the relay taught alone first; lookup never formed on this seed | 1.326 | 100.0% | 1.4% |
| `exealm_together_gated_seed0.pt` | the same recipe, with gated attention and QK-norm from Qwen3.5 (18.49M parameters) | 1.303 | 100.0% | 93.1% |
| `exealm_together_search_seed0.pt` | the same recipe, with TONN's circuit search on for the first half | 1.333 | 100.0% | 94.4% |
| `exealm_recipe1_seed0.pt` | ExeaLM, 80% text, board posts every 64 tokens; neither skill formed | 1.294 | 1.0% | |
| `llama_samebatches_seed0.pt`, `_seed1.pt` | Llama on the same batches as ExeaLM | 1.339, 1.332 | | |
| `llama_samebatches_sametime_seed0.pt` | Llama given ExeaLM's training time (31,200 steps) | 1.314 | | |
| `llama_textonly_seed0.pt` | Llama on text and Wikipedia only | 1.288 | | |
| `llama_recipe1_seed0.pt` | Llama on the batches of `exealm_recipe1` | 1.294 | | |

`finetunes/` holds 2,000 to 8,000-step fine-tunes of `exealm_together_seed0.pt` that practice memory on real text
(issue [#27](https://github.com/exea-labs/ExeaLM/issues/27)). `checks/` holds short runs (2,000 steps or fewer) from
the training-schedule tests. The relay-first seed-0 model was overwritten before checkpoints carried the seed; its
step-4,000 checkpoint is in `checks/`.

`checkpoints/100m/exealm_together_100m_step3000_seed0.pt` is the 0.1B ExeaLM (90.65M parameters, the `100m` preset)
trained with the same recipe and stopped at step 3,000 of the 24,000-step schedule (49M tokens), so that it finished
before we lost the Mac Studio. It is
under-trained: English 1.790 bits per byte, relay 99.1%, lookup 73.1% (still rising: 52.6% at step 1,000, 67.3% at
2,000). It loads the same way as the 17M files.

Three more 0.1B checkpoints come from one session on an AMD MI300X (15 and 16 November 2026), each stopped at step
3,000 (49M tokens) and under-trained in the same way:

| File in `checkpoints/100m/` | What it is | English, bits per byte | Relay | Lookup |
|---|---|---|---|---|
| `exealm_combined_wakey_step3000_seed0.pt` | ExeaLM (94.7M) with the board, the library and a weight-alter slot in every TONN layer (`wa_key="hidden"`) | 1.726 | 100.0% | 75.0% |
| `llama_combined_step3000_seed0.pt` | Llama (89.7M) on the same batches, learning rate 2e-3 | 2.046 | 1.4% | 1.5% |
| `llama_combined_lr1e3_step3000_seed0.pt` | the same at learning rate 1e-3 | 2.013 | 1.8% | 1.5% |

In the ExeaLM file the relay is carried by weight edits alone (100% with the board off, 0.0% with the weight-alter
slots off). This recipe trained in 2 of 5 ExeaLM runs at this size; the other runs are in `results/combined/`. The
same-size comparisons without tasks (ExeaLM 1.413 to 1.427, gated Llama 1.446, Qwen3.5-style 1.457, Llama 1.509
bits per byte, 3 seeds each) saved no checkpoints; their tables are in `results/funding/README.md`.

```python
import exealm                             # pip install -e . in a clone of github.com/exea-labs/ExeaLM, and huggingface_hub
model, tok = exealm.from_pretrained("17m/exealm_together_gated_seed0")    # any file under checkpoints/, without ".pt"
```

That downloads the checkpoint and the tokenizer from this repo and works for the `llama_*` files too. To write with
a checkpoint in the terminal, alone or as a swarm of agents that share a board:

```bash
python -m exealm.chat --ckpt 17m/exealm_together_gated_seed0 --raw --swarm 4
```

These are base models. They continue text; they have not been through post-training and do not chat yet.

To watch the two skills: `python experiments/show_skills.py --ckpt <file>`. The real-text memory test:
`python experiments/irl_eval.py --ckpt <file>`.

## Files

- `checkpoints/17m/`: the research checkpoints above.
- `checkpoints/100m/`: the under-trained 0.1B checkpoints above.
- `results/`: every result file and log behind the numbers on this page and in the GitHub report.
- `tokenizer/tokenizer.json`: the 32k byte-level BPE tokenizer.
  - **Trained on:** 200 MB of English (FineWeb-Edu), 160 MB of Japanese (LLM-jp Corpus v4: Wikipedia and FineWeb-2)
    and 80 MB of code.
  - **Why 32k:** it won or tied against 49k and 64k at a fixed token budget.
  - **Special tokens:** chat roles, plus `<|retrieved|>` and `<|board|>` for the memory circuits.

```python
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer/tokenizer.json")
```

## Intended use

Research and education on how open language models are built. Not for high-stakes decisions.

## Limitations

No evaluation results on standard benchmarks yet; the numbers above are language-modeling loss on small held-out sets.
Like other web-scale models, future checkpoints may produce incorrect, biased, or harmful text. We will publish evals
and known limitations with each checkpoint.

## Contributing

Anyone can contribute through GitHub. See [CONTRIBUTING.md](https://github.com/exea-labs/ExeaLM/blob/main/CONTRIBUTING.md)
and the [Code of Conduct](https://github.com/exea-labs/ExeaLM/blob/main/CODE_OF_CONDUCT.md).