Nano-GLM re-implements the architecture behind modern GLM and DeepSeek-class models at a scale that trains on one consumer GPU — multi-latent attention, mixture-of-experts routing, and sparse attention, none of it simplified.
Instead of caching a full key/value tensor per token, MLA compresses attention through a low-rank projection, closer to LoRA than to a standard cache. The memory saved is what makes training possible on 6GB of VRAM at all.
Rather than one dense feed-forward block, the model routes each token to a small set of specialized experts plus one shared expert. Capacity grows without every token paying for every parameter.
A learned indexer decides which previous tokens are actually worth attending to, and drops the rest — attention over a shortlist instead of the full context, without giving up long-range dependencies.