|
Download README.md from exea-labs/ExeaLM: direct link, hf CLI and curl.
- Browser
- Download file 9.77 kB
-
https://huggingface.co/exea-labs/ExeaLM/resolve/main/README.md
- Command line
-
hf download hf://exea-labs/ExeaLM/README.md
-
curl -L -o README.md https://huggingface.co/exea-labs/ExeaLM/resolve/main/README.md
9.77 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| - ja | |
| tags: | |
| - exealm | |
| - open-source | |
| - from-scratch | |
| - research-preview | |
| - tonn | |
| - hybrid | |
| - retrieval | |
| - multi-agent | |
| # ExeaLM | |
| An open language model built by high schoolers, run by [Exea Labs](https://exealabs.org). | |
| **Status: research preview.** The planned 0.1B model is not trained yet. This repo holds the tokenizer, the | |
| research checkpoints of the 17M-parameter experiments (`checkpoints/17m/`, below) and every result file | |
| (`results/`). Everything is reproducible from | |
| [github.com/exea-labs/ExeaLM](https://github.com/exea-labs/ExeaLM); the full report is | |
| [results/README.md](https://github.com/exea-labs/ExeaLM/blob/main/results/README.md). | |
| ## Links | |
| - Code: https://github.com/exea-labs/ExeaLM | |
| - Contributor sign-up: https://docs.google.com/forms/d/e/1FAIpQLSdPHVZhuXxBDUf-cJM2zDbj1JZPsduu95Y_kTJ_g4gKAXdDVw/viewform | |
| - Architecture paper: [TONN: Networks That Build Their Own Architecture from Data](https://www.preprints.org/manuscript/202611.0994) | |
| ## Model (planned) | |
| - **Size:** 0.1 billion parameters (105M with the planned vocabulary, about 94M with this 32k tokenizer). | |
| - **Architecture:** a hybrid of TONN and attention. Every third layer is a TONN layer: small learned circuits | |
| (convolutions, running sums, state spaces, memories) whose layout TONN's search adjusts during training. The other | |
| layers are multi-head attention with two parts from Qwen3.5: gated attention and QK-norm. Each layer is followed by | |
| a gated MLP. | |
| - **Built-in memory circuits** in every TONN layer: | |
| - retrieval from a read-only library, with the model's own embeddings; | |
| - a read-write message board shared by several copies of the model, so agents can talk. | |
| - **Training data:** about 100B tokens; see [DATA.md](https://github.com/exea-labs/ExeaLM/blob/main/DATA.md). | |
| - **License:** Apache 2.0 for code and weights. | |
| ## Results so far (small-scale, on one Mac Studio GPU) | |
| Bits per byte on held-out English, Japanese and code (lower is better). | |
| | Size | Comparison | ExeaLM hybrid | Same-size Llama-style transformer | | |
| |---|---|---|---| | |
| | 13M | same training steps, 2 seeds | **1.531** | 1.554 | | |
| | 27M | same training steps, best learning rates, 2 seeds | **1.492** | 1.523 | | |
| | 48M | same training time, 2 seeds | **1.445** | 1.452 | | |
| | ~0.1B (90M) | same training time, 2 seeds | **1.455** | 1.479 | | |
| | 13M, with gated attention and QK-norm (from Qwen3.5) | same training time, 1 seed | **1.509** | 1.542 (1.515 with the same additions) | | |
| - At 0.1B the hybrid trains at the same speed as the transformer (1.01 times its cost per token). | |
| - One 18M model trained mostly on real text also passes facts between agents through its board (100% correct, 1% | |
| with the board cut) and looks facts up in a library it has never seen (94%, 1.6% without it), on both seeds. These | |
| are synthetic tasks. | |
| - Gated attention and QK-norm also help the 18M model with both skills: 1.393 bits per byte averaged over the four | |
| held-out sets, against 1.419 without them (equal steps, one seed). It keeps the relay (100%) and lookup (93%). | |
| - On real text the memory circuits do not pay off yet. On held-out Wikipedia, putting the right article in the | |
| context gains about 1 bit per re-mentioned rare word; the model's library and board gain between -0.1 and +0.06. | |
| The board path is also fragile: after 8,000 steps of practice on single articles, notes from all 256 test articles | |
| at once make re-mentions 7.9 bits worse. | |
| - **Correction:** we reported that training as a swarm beats a matched control by 0.014 bits per byte. Reruns range | |
| from 0.015 better to 0.035 worse, so the swarm gain on real text is unproven. | |
| All of these are short runs (about 50M training tokens), far smaller than the planned 100B-token run. | |
| ## Research checkpoints (`checkpoints/17m/`) | |
| Small models from the experiments in | |
| [results/README.md](https://github.com/exea-labs/ExeaLM/blob/main/results/README.md), sections 10 to 12. Each was trained | |
| for 24,000 steps of 32 × 512 tokens (393M tokens) on one Mac Studio. **They are research artifacts, not a release:** | |
| small, briefly trained, and not evaluated on any benchmark. | |
| The ExeaLM checkpoints are the `small` hybrid at width 320: 6 layers, TONN layers at 0 and 3, each with a retrieval | |
| circuit and an agent-board circuit (18.08M parameters). The Llama checkpoints are a same-size Llama-style transformer | |
| (17.83M). Scores: English bits per byte (lower is better); relay through the board and lookup in an unseen library | |
| (chance is 1.6%). | |
| | File | Model and training | English | Relay | Lookup | | |
| |---|---|---|---|---| | |
| | `exealm_together_seed0.pt` | ExeaLM, both skills taught together from step 0 (the recipe to use) | 1.320 | 100.0% | 94.0% | | |
| | `exealm_together_seed1.pt` | the same, seed 1 | 1.328 | 100.0% | 94.3% | | |
| | `exealm_relayfirst_seed1.pt` | ExeaLM, the relay taught alone first; lookup never formed on this seed | 1.326 | 100.0% | 1.4% | | |
| | `exealm_together_gated_seed0.pt` | the same recipe, with gated attention and QK-norm from Qwen3.5 (18.49M parameters) | 1.303 | 100.0% | 93.1% | | |
| | `exealm_together_search_seed0.pt` | the same recipe, with TONN's circuit search on for the first half | 1.333 | 100.0% | 94.4% | | |
| | `exealm_recipe1_seed0.pt` | ExeaLM, 80% text, board posts every 64 tokens; neither skill formed | 1.294 | 1.0% | | | |
| | `llama_samebatches_seed0.pt`, `_seed1.pt` | Llama on the same batches as ExeaLM | 1.339, 1.332 | | | | |
| | `llama_samebatches_sametime_seed0.pt` | Llama given ExeaLM's training time (31,200 steps) | 1.314 | | | | |
| | `llama_textonly_seed0.pt` | Llama on text and Wikipedia only | 1.288 | | | | |
| | `llama_recipe1_seed0.pt` | Llama on the batches of `exealm_recipe1` | 1.294 | | | | |
| `finetunes/` holds 2,000 to 8,000-step fine-tunes of `exealm_together_seed0.pt` that practice memory on real text | |
| (issue [#27](https://github.com/exea-labs/ExeaLM/issues/27)). `checks/` holds short runs (2,000 steps or fewer) from | |
| the training-schedule tests. The relay-first seed-0 model was overwritten before checkpoints carried the seed; its | |
| step-4,000 checkpoint is in `checks/`. | |
| `checkpoints/100m/exealm_together_100m_step3000_seed0.pt` is the 0.1B ExeaLM (90.65M parameters, the `100m` preset) | |
| trained with the same recipe and stopped at step 3,000 of the 24,000-step schedule (49M tokens), so that it finished | |
| before we lost the Mac Studio. It is | |
| under-trained: English 1.790 bits per byte, relay 99.1%, lookup 73.1% (still rising: 52.6% at step 1,000, 67.3% at | |
| 2,000). It loads the same way as the 17M files. | |
| Three more 0.1B checkpoints come from one session on an AMD MI300X (15 and 16 November 2026), each stopped at step | |
| 3,000 (49M tokens) and under-trained in the same way: | |
| | File in `checkpoints/100m/` | What it is | English, bits per byte | Relay | Lookup | | |
| |---|---|---|---|---| | |
| | `exealm_combined_wakey_step3000_seed0.pt` | ExeaLM (94.7M) with the board, the library and a weight-alter slot in every TONN layer (`wa_key="hidden"`) | 1.726 | 100.0% | 75.0% | | |
| | `llama_combined_step3000_seed0.pt` | Llama (89.7M) on the same batches, learning rate 2e-3 | 2.046 | 1.4% | 1.5% | | |
| | `llama_combined_lr1e3_step3000_seed0.pt` | the same at learning rate 1e-3 | 2.013 | 1.8% | 1.5% | | |
| In the ExeaLM file the relay is carried by weight edits alone (100% with the board off, 0.0% with the weight-alter | |
| slots off). This recipe trained in 2 of 5 ExeaLM runs at this size; the other runs are in `results/combined/`. The | |
| same-size comparisons without tasks (ExeaLM 1.413 to 1.427, gated Llama 1.446, Qwen3.5-style 1.457, Llama 1.509 | |
| bits per byte, 3 seeds each) saved no checkpoints; their tables are in `results/funding/README.md`. | |
| ```python | |
| import exealm # pip install -e . in a clone of github.com/exea-labs/ExeaLM, and huggingface_hub | |
| model, tok = exealm.from_pretrained("17m/exealm_together_gated_seed0") # any file under checkpoints/, without ".pt" | |
| ``` | |
| That downloads the checkpoint and the tokenizer from this repo and works for the `llama_*` files too. To write with | |
| a checkpoint in the terminal, alone or as a swarm of agents that share a board: | |
| ```bash | |
| python -m exealm.chat --ckpt 17m/exealm_together_gated_seed0 --raw --swarm 4 | |
| ``` | |
| These are base models. They continue text; they have not been through post-training and do not chat yet. | |
| To watch the two skills: `python experiments/show_skills.py --ckpt <file>`. The real-text memory test: | |
| `python experiments/irl_eval.py --ckpt <file>`. | |
| ## Files | |
| - `checkpoints/17m/`: the research checkpoints above. | |
| - `checkpoints/100m/`: the under-trained 0.1B checkpoints above. | |
| - `results/`: every result file and log behind the numbers on this page and in the GitHub report. | |
| - `tokenizer/tokenizer.json`: the 32k byte-level BPE tokenizer. | |
| - **Trained on:** 200 MB of English (FineWeb-Edu), 160 MB of Japanese (LLM-jp Corpus v4: Wikipedia and FineWeb-2) | |
| and 80 MB of code. | |
| - **Why 32k:** it won or tied against 49k and 64k at a fixed token budget. | |
| - **Special tokens:** chat roles, plus `<|retrieved|>` and `<|board|>` for the memory circuits. | |
| ```python | |
| from tokenizers import Tokenizer | |
| tok = Tokenizer.from_file("tokenizer/tokenizer.json") | |
| ``` | |
| ## Intended use | |
| Research and education on how open language models are built. Not for high-stakes decisions. | |
| ## Limitations | |
| No evaluation results on standard benchmarks yet; the numbers above are language-modeling loss on small held-out sets. | |
| Like other web-scale models, future checkpoints may produce incorrect, biased, or harmful text. We will publish evals | |
| and known limitations with each checkpoint. | |
| ## Contributing | |
| Anyone can contribute through GitHub. See [CONTRIBUTING.md](https://github.com/exea-labs/ExeaLM/blob/main/CONTRIBUTING.md) | |
| and the [Code of Conduct](https://github.com/exea-labs/ExeaLM/blob/main/CODE_OF_CONDUCT.md). | |