tlm26 — Mixture-of-Experts scaling family

Sparse Mixture-of-Experts language models trained on TinyStories, part of the tlm26 project (sub-1M-parameter LMs for microcontroller deployment; USC ANRG). This repository holds the MoE expert-scaling family: a shared backbone with the expert count swept from 8 to 128, each trained to a common 30,000-step budget.

Expert scaling

All models share a d=64, 5-layer backbone (8 heads, 4 KV heads, vocab 512, seq 256) with top-2 gating. Only the number of experts varies. Bits-per-byte is measured on the held-out shard.

Experts	CE	bits/byte	Total params	Active params/token
8	1.0153	0.7050	1.57M	466K
16	0.9474	0.6579	3.05M	469K
32	0.8811	0.6119	6.00M	474K
64	0.8256	0.5733	11.9M	484K
128	0.7872	0.5466	23.7M	505K

Each doubling of experts lowers bits/byte while active parameters per token stay near-constant (466K → 505K): added sparse capacity improves quality at almost fixed inference compute. Total parameters grow with the expert count, which sets the flash footprint for on-device deployment.

A top-1 variant of the 8-expert model (moe_d64_e8_top1) is included for comparison.

Files

Each moe_d64_e{N}/model.pt is a weights-only checkpoint (optimizer state stripped) holding the model state dict and model_args.

Loading

import torch
ck = torch.load("moe_d64_e128/model.pt", map_location="cpu")
args = ck["model_args"]        # dim, n_layers, n_experts, moe_top_k, ...
state = ck["model"]            # load into the tlm26 MoE model

Model definition and training/eval code: https://github.com/ANRGUSC/tlm26