ExeaLM

An open language model built by high schoolers, run by Exea Labs.

Status: research preview. The planned 0.1B model is not trained yet. This repo holds the tokenizer, the research checkpoints of the 17M-parameter experiments (checkpoints/17m/, below) and every result file (results/). Everything is reproducible from github.com/exea-labs/ExeaLM; the full report is results/README.md.

Links

• Code: https://github.com/exea-labs/ExeaLM
• Contributor sign-up: https://docs.google.com/forms/d/e/1FAIpQLSdPHVZhuXxBDUf-cJM2zDbj1JZPsduu95Y_kTJ_g4gKAXdDVw/viewform
• Architecture paper: TONN: Networks That Build Their Own Architecture from Data

Model (planned)

• Size: 0.1 billion parameters (101M with the planned vocabulary, about 90M with this 32k tokenizer).
• Architecture: a hybrid of TONN and attention. Every third layer is a TONN layer: small learned circuits (convolutions, running sums, state spaces, memories) whose layout TONN's search adjusts during training. The other layers are standard multi-head attention. Each layer is followed by a gated MLP.
• Built-in memory circuits in every TONN layer:
  • retrieval from a read-only library, with the model's own embeddings;
  • a read-write message board shared by several copies of the model, so agents can talk.
• Training data: about 100B tokens; see DATA.md.
• License: Apache 2.0 for code and weights.

Results so far (small-scale, on one Mac Studio GPU)

Bits per byte on held-out English, Japanese and code (lower is better).

Size	Comparison	ExeaLM hybrid	Same-size Llama-style transformer
13M	same training steps, 2 seeds	1.531	1.554
27M	same training steps, best learning rates, 2 seeds	1.492	1.523
48M	same training time, 2 seeds	1.445	1.452
~0.1B (90M)	same training time, 2 seeds	1.455	1.479
13M, with gated attention and QK-norm (from Qwen3.5)	same training time, 1 seed	1.509	1.542 (1.515 with the same additions)

• At 0.1B the hybrid trains at the same speed as the transformer (1.01 times its cost per token).
• One 18M model trained mostly on real text also passes facts between agents through its board (100% correct, 1% with the board cut) and looks facts up in a library it has never seen (94%, 1.6% without it), on both seeds. These are synthetic tasks.
• Gated attention and QK-norm also help the 18M model with both skills: 1.393 bits per byte averaged over the four held-out sets, against 1.419 without them (equal steps, one seed). It keeps the relay (100%) and lookup (93%).
• On real text the memory circuits do not pay off yet. On held-out Wikipedia, putting the right article in the context gains about 1 bit per re-mentioned rare word; the model's library and board gain between -0.1 and +0.06. The board path is also fragile: after 8,000 steps of practice on single articles, notes from all 256 test articles at once make re-mentions 7.9 bits worse.
• Correction: we reported that training as a swarm beats a matched control by 0.014 bits per byte. Reruns range from 0.015 better to 0.035 worse, so the swarm gain on real text is unproven.

All of these are short runs (about 50M training tokens), far smaller than the planned 100B-token run.

Research checkpoints (checkpoints/17m/)

Small models from the experiments in results/README.md, sections 10 to 12. Each was trained for 24,000 steps of 32 × 512 tokens (393M tokens) on one Mac Studio. They are research artifacts, not a release: small, briefly trained, and not evaluated on any benchmark.

The ExeaLM checkpoints are the small hybrid at width 320: 6 layers, TONN layers at 0 and 3, each with a retrieval circuit and an agent-board circuit (18.08M parameters). The Llama checkpoints are a same-size Llama-style transformer (17.83M). Scores: English bits per byte (lower is better); relay through the board and lookup in an unseen library (chance is 1.6%).

File	Model and training	English	Relay	Lookup
exealm_together_seed0.pt	ExeaLM, both skills taught together from step 0 (the recipe to use)	1.320	100.0%	94.0%
exealm_together_seed1.pt	the same, seed 1	1.328	100.0%	94.3%
exealm_relayfirst_seed1.pt	ExeaLM, the relay taught alone first; lookup never formed on this seed	1.326	100.0%	1.4%
exealm_together_gated_seed0.pt	the same recipe, with gated attention and QK-norm from Qwen3.5 (18.49M parameters)	1.303	100.0%	93.1%
exealm_together_search_seed0.pt	the same recipe, with TONN's circuit search on for the first half	1.333	100.0%	94.4%
exealm_recipe1_seed0.pt	ExeaLM, 80% text, board posts every 64 tokens; neither skill formed	1.294	1.0%
llama_samebatches_seed0.pt, _seed1.pt	Llama on the same batches as ExeaLM	1.339, 1.332
llama_samebatches_sametime_seed0.pt	Llama given ExeaLM's training time (31,200 steps)	1.314
llama_textonly_seed0.pt	Llama on text and Wikipedia only	1.288
llama_recipe1_seed0.pt	Llama on the batches of exealm_recipe1	1.294

finetunes/ holds 2,000 to 8,000-step fine-tunes of exealm_together_seed0.pt that practice memory on real text (issue #27). checks/ holds short runs (2,000 steps or fewer) from the training-schedule tests. The relay-first seed-0 model was overwritten before checkpoints carried the seed; its step-4,000 checkpoint is in checks/.

checkpoints/100m/exealm_together_100m_step3000_seed0.pt is the 0.1B ExeaLM (90.65M parameters, the 100m preset) trained with the same recipe and stopped at step 3,000 of the 24,000-step schedule (49M tokens), so that it finished before we lost the Mac Studio. It is under-trained: English 1.790 bits per byte, relay 99.1%, lookup 73.1% (still rising: 52.6% at step 1,000, 67.3% at 2,000). It loads the same way as the 17M files.

import torch
from exealm import ExeaLM                  # pip install -e . in a clone of github.com/exea-labs/ExeaLM
from exealm.config import ModelConfig
from exealm.model.baseline import Llama

ck = torch.load("checkpoints/17m/exealm_together_seed0.pt", map_location="cpu", weights_only=False)
cfg = ModelConfig.from_dict(ck["cfg"])
model = ExeaLM(cfg, ck["specs"])          # for a llama_*.pt file: model = Llama(cfg)
model.load_state_dict(ck["model"])
model.encoder.idf_ready = True            # keep the retrieval encoder's word rarity from training

To watch the two skills: python experiments/show_skills.py --ckpt <file>. The real-text memory test: python experiments/irl_eval.py --ckpt <file>.

Files

• checkpoints/17m/: the research checkpoints above.
• checkpoints/100m/: the under-trained 0.1B checkpoint above.
• results/: every result file and log behind the numbers on this page and in the GitHub report.
• tokenizer/tokenizer.json: the 32k byte-level BPE tokenizer.
  • Trained on: 200 MB of English (FineWeb-Edu), 160 MB of Japanese (LLM-jp Corpus v4: Wikipedia and FineWeb-2) and 80 MB of code.
  • Why 32k: it won or tied against 49k and 64k at a fixed token budget.
  • Special tokens: chat roles, plus <|retrieved|> and <|board|> for the memory circuits.

from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer/tokenizer.json")

Intended use

Research and education on how open language models are built. Not for high-stakes decisions.

Limitations

No evaluation results on standard benchmarks yet; the numbers above are language-modeling loss on small held-out sets. Like other web-scale models, future checkpoints may produce incorrect, biased, or harmful text. We will publish evals and known limitations with each checkpoint.

Contributing

Anyone can contribute through GitHub. See CONTRIBUTING.md and the Code of Conduct.