Edge0/Edge0-35B-A3B-preview

Edge0-35b-a3b Preview

A 35B-class sparse MoE that runs on a phone — 2.9 GiB of active memory, experts streamed from SSD.

Edge0-35b-a3b — an 35B MoE LLM that runs at viable speed on portable devices in under 2.9 GiB of active memory (1/8 of its 23 GB weight footprint), via the edge0 streaming inference framework.

The key is streaming: experts are memory-mapped and fetched from SSD only as routed, so RAM holds just the active weights. What makes that viable — instead of stalling like plain parameter offloading — is a trained prerouter head that predicts the next token's expert routing one step ahead, hiding storage latency behind compute.

Preview status: this is an early preview release of the edge0 pipeline. The checkpoint ships as int4 quantization plus LoRA and prerouter adapters trained for this framework.

Model summary

Base model	Qwen3.5-MoE 35B-A3B
Quantization	4-bit
Layers	40
Experts / active per token	256 / 4 (K=4)
Framework	edge0 (MLX backend)
Contents	base checkpoint + lora_edge0_35b.safetensors + prerouter_edge0_35b.safetensors

The LoRA and prerouter adapters are co-located with the base checkpoint and load automatically — this repository is a complete, ready-to-run model directory for edge0.

Quality

All benchmarks were run by us with OpenCompass under identical settings and parameters for both models. The loss of the edge0 pipeline (int4 + adapters) relative to the fp16 base model is small: 3.9 points on average (max 100):

Benchmark	edge0-35b (int4)	Qwen3.5-MoE 35B-A3B (fp16)
AIME 2026	86.6	92.7
HumanEval	90.9	95.1
GPQA-Diamond	79.8	81.8
MMLU-Pro	81.0	84.6
IFBench	57.9	61.7
Average	79.2	83.2

Performance

Measured with examples/bench.py on a Mac mini M4 Pro, 24 GB:

Decode speed	Prefill throughput (cold / warm)	Peak active memory*
14.9–17.7 tok/s	113 / 140 tok/s	2.9 GiB

*Short contexts; long contexts add KV cache. Expert weights stream from SSD via mmap and are not resident.

Quick start

pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'

# Download this repository into a local directory
huggingface-cli download Edge0/Edge0-35b-a3b-preview --local-dir ./Edge0-35b-a3b-preview

# Run it
export EDGE0_35B_MODEL=$PWD/Edge0-35b-a3b-preview
edge0 chat --name edge0-35b --prompt "Introduce yourself"

# Or serve an OpenAI-compatible HTTP API
edge0 serve --name edge0-35b --port 8085

For full usage (Python API, streaming options, prerouter details), see the edge0 documentation.

License

Apache 2.0. See LICENSE.