license: mit
library_name: pytorch
tags:
- mixture-of-experts
- tiny-language-model
- tinystories
- text-generation
pipeline_tag: text-generation
tlm26 — Mixture-of-Experts scaling family
Sparse Mixture-of-Experts language models trained on TinyStories, part of the tlm26 project (sub-1M-parameter LMs for microcontroller deployment; USC ANRG). This repository holds the MoE expert-scaling family: a shared backbone with the expert count swept from 8 to 128, each trained to a common 30,000-step budget.
Expert scaling
All models share a d=64, 5-layer backbone (8 heads, 4 KV heads, vocab 512,
seq 256) with top-2 gating. Only the number of experts varies. Bits-per-byte
is measured on the held-out shard.
| Experts | CE | bits/byte | Total params | Active params/token |
|---|---|---|---|---|
| 8 | 1.0153 | 0.7050 | 1.57M | 466K |
| 16 | 0.9474 | 0.6579 | 3.05M | 469K |
| 32 | 0.8811 | 0.6119 | 6.00M | 474K |
| 64 | 0.8256 | 0.5733 | 11.9M | 484K |
| 128 | 0.7872 | 0.5466 | 23.7M | 505K |
Each doubling of experts lowers bits/byte while active parameters per token stay near-constant (466K → 505K): added sparse capacity improves quality at almost fixed inference compute. Total parameters grow with the expert count, which sets the flash footprint for on-device deployment.
A top-1 variant of the 8-expert model (moe_d64_e8_top1) is included for
comparison.
Files
Each moe_d64_e{N}/model.pt is a weights-only checkpoint (optimizer state
stripped) holding the model state dict and model_args.
Loading
import torch
ck = torch.load("moe_d64_e128/model.pt", map_location="cpu")
args = ck["model_args"] # dim, n_layers, n_experts, moe_top_k, ...
state = ck["model"] # load into the tlm26 MoE model
Model definition and training/eval code: https://github.com/ANRGUSC/tlm26