moe-l2 — Run 100B+ MoE models on low-VRAM NVIDIA GPUs
moe-l2 is an expert-offload scheduler for llama.cpp that lets you run large MoE (Mixture-of-Experts) models — DeepSeek, Qwen, Mixtral, 100B+ params — on consumer NVIDIA GPUs with far less VRAM than the model would normally need.
Why
MoE models are like a hospital: every token only activates a few experts (the "doctors on duty"), but all experts must stay "on shift" (loaded into VRAM). A 16B MoE model with a 6 GB file can require 23.3 GB of VRAM when fully loaded — an 8 GB card can't even start it.
moe-l2 keeps the experts in host RAM (zero VRAM) and only moves the activated experts to the GPU per step, with an LRU hot-expert cache. The GPU reads pinned expert weights directly via PCIe DMA.
Measured results (RTX 4090, full chain moe-l2 start --gpu)
| Model | Full-load VRAM | With moe-l2 | Speed |
|---|---|---|---|
| DeepSeek-V2-Lite (16B MoE) | 23.3 GB | 1.6-2.0 GB | 139-154 t/s |
| Qwen3.6-A3B (32B MoE) | 7.6 GB+ | 2.9 GB | 50.2 t/s (single) |
| DeepSeek-V4-Flash (157B MoE, 85 GB file) | OOM | 8.3-9.1 GB VRAM | N/A (upstream bug) |
Three-card full-chain measurements (2026-08-19, bins-v0.6.0):
| GPU | DeepSeek-V2-Lite | Qwen3.6-A3B |
|---|---|---|
| RTX 4090 | 139-154 t/s | 25.5-44.2 t/s |
| RTX 2080 Ti (11 GB) | 86-94 t/s | 16.6-28.6 t/s |
| RTX 5090 | 141-151 t/s | 28-52.5 t/s |
How it works (3 layers)
- Domain predictor — classifies your prompt's domain (8 domains) and preloads the most likely experts.
- L2 cache — mmap shared-memory LRU cache, async preload, zero-copy GGUF reading.
- A3 GPU expert cache — per-slot locked LRU in VRAM; hot experts stay resident, cold experts are evicted. Router-map driven selective pin keeps only top-K experts per layer locked, so an 85 GB model runs at ~10.4 GB RSS with zero speed regression.
Transparent OpenAI-compatible proxy on localhost:11435 (SSE streaming), so it works with Open WebUI, LangChain, etc.
Quick start
# Linux x86_64 + NVIDIA GPU + CUDA driver
pip install moe-l2
moe-l2 download-bins
moe-l2 start --model /path/to/model.gguf --gpu
moe-l2 doctor checks your environment; moe-l2 model download fetches models with resumable downloads.
Requirements
- Linux x86_64
- NVIDIA GPU (GTX 1080 → RTX 50, one prebuilt binary for all, sm_61-sm_120a)
- Python 3.9+
- Native Windows / macOS / ARM Linux not supported (WSL2 unverified)
Version history
13 versions in one month: 23.3 GB → 1.2 GB VRAM, from "can't run" to 140+ t/s. Full changelog on GitHub.
Links
- Source code: github.com/yalun753/moe-l2
- PyPI: pypi.org/project/moe-l2
- Benchmarks & verification reports: in the GitHub
references/directory (EN/ZH) - White paper: white-paper
License
Apache-2.0