license: fair-noncommercial-research-license
extra_gated_fields:
First Name: text
Last Name: text
Date of birth: date_picker
Country: country
Affiliation: text
Job title:
type: select
options:
- Student
- Research Graduate
- AI researcher
- AI developer/engineer
- Reporter
- Other
geo: ip_location
By clicking Submit below I accept the terms of the license and acknowledge that the information I provide will be collected stored processed and shared in accordance with the Meta Privacy Policy: checkbox
extra_gated_description: >-
The information you provide will be collected, stored, processed and shared in
accordance with the [Meta Privacy
Policy](https://www.facebook.com/privacy/policy/).
extra_gated_button_content: Submit
language:
- en
library_name: transformers
tags:
- facebook
- meta
- pytorch
- mixture-of-experts
- MoE
- on-device
MobileMoE-L (SFT) Model Card
MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-L-SFT repository — the instruction-tuned 0.9B-active model.
| S | M | L | |
|---|---|---|---|
| Active / total params | 272M / 1.3B | 528M / 2.8B | 922M / 5.3B |
| Layers | 20 | 26 | 32 |
| Model dimension | 768 | 1024 | 1280 |
| Heads (Q / KV) | 12 / 4 | 16 / 4 | 20 / 4 |
| Routed experts | 60 | 60 | 60 |
| Top-k | 4 | 4 | 4 |
| INT4 weight memory | 0.68 GB | 1.48 GB | 2.75 GB |
| Base | MobileMoE-S-Base | MobileMoE-M-Base | MobileMoE-L-Base |
| SFT | MobileMoE-S-SFT | MobileMoE-M-SFT | MobileMoE-L-SFT |
| QAT (INT4) | MobileMoE-S-QAT | MobileMoE-M-QAT | MobileMoE-L-QAT |
For the detailed technical report: 📝 MobileMoE: Scaling On-Device Mixture of Experts
For more versions, check out the 🤗 MobileMoE Collection
MobileMoE establishes a new Pareto frontier for on-device LLMs. Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute Finf = 2Nact (GFLOPs) and (b) total parameters Ntotal (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB). Accuracy is shown for the instruction-tuned models.
Key Features
- A new Pareto frontier for on-device LLMs. Across 14 foundational benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs at 2–4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters.
- Scaling-law-derived architecture. The architecture is derived from an on-device MoE scaling law that jointly optimizes under mobile memory and compute constraints, identifying an on-device sweet spot: moderate sparsity, with fine-grained experts and shared expert.
- Four-stage recipe. Pre-training → mid-training → instruction fine-tuning → INT4 quantization-aware training, all on open-source datasets.
Model Information
Model: MobileMoE-L-SFT (instruction-tuned)
Active Parameters: 922M
Total Parameters: 5.3B
Layers: 32
Model Dimension: 1280
Attention Heads: 20
KV Heads: 4 (GQA)
Head Dimension: 64
Routed Experts: 60 (fine-grained, FFN hidden dim 640 each)
Active Experts per Token: 4 (top-k sigmoid routing, with normalization)
Shared Expert: 1, always on (FFN hidden dim 2560)
Vocabulary Size: 128,256
Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)
Chat Template: Yes (end-of-turn token <|eot|>)
Input Modality: Text
Output Modality: Text
Languages: English
Training Stages: Pre-training → mid-training → supervised fine-tuning (SFT)
Context Length: 8,192 tokens
Precision: BF16
Model Developer: Meta
Model Release Date: Aug 2026
License: MobileMoE is FAIR NC licensed
Results
All results below are for the instruction-tuned (SFT) models. We re-evaluated every model under identical settings in non-thinking mode with greedy decoding, using lm-eval together with the official allenai/IFBench package. Few-shot counts are shown in parentheses after benchmark names; benchmarks without a count are evaluated 0-shot. The MobileMoE results use the exact weights in this repository, which include brief fine-tuning with self-identity beyond the SFT checkpoint in the technical report, resulting in a small difference: a foundational-benchmark average of 60.3 here versus 60.1 in the report.
Foundational benchmarks
| Capability | Benchmark | MobileLLM-Pro | OLMo 2 1B | SmolLM2 1.7B | Qwen3.5 2B | OLMoE-1B-7B | MobileMoE-L |
|---|---|---|---|---|---|---|---|
| Active / total params | 1.1B | 1.5B | 1.7B | 1.9B | 1.3B / 6.9B | 922M / 5.3B | |
| Commonsense Reasoning | HellaSwag | 52.2 | 67.3 | 71.7 | 62.2 | 78.8 | 72.9 |
| PIQA | 73.2 | 75.2 | 76.2 | 72.8 | 79.7 | 79.4 | |
| SIQA | 42.7 | 46.1 | 44.6 | 41.0 | 50.8 | 53.6 | |
| WinoGrande | 51.7 | 63.5 | 68.4 | 63.0 | 68.7 | 67.0 | |
| Knowledge | MMLU (5-shot) | 38.7 | 42.9 | 49.4 | 57.4 | 52.7 | 60.3 |
| NaturalQuestions (5-shot) | 8.9 | 12.4 | 14.2 | 8.8 | 17.2 | 22.1 | |
| TriviaQA (5-shot) | 19.9 | 37.6 | 46.0 | 28.1 | 54.1 | 55.5 | |
| Science | ARC-Challenge (25-shot) | 35.8 | 45.1 | 53.4 | 53.2 | 57.6 | 57.8 |
| ARC-Easy | 52.7 | 69.8 | 62.9 | 66.0 | 75.9 | 82.3 | |
| OpenBookQA | 29.6 | 42.2 | 45.8 | 35.2 | 46.8 | 43.0 | |
| Reading | BoolQ | 68.8 | 71.0 | 68.5 | 71.7 | 81.1 | 81.7 |
| DROP (3-shot) | 25.5 | 31.2 | 24.8 | 44.7 | 29.3 | 51.3 | |
| Reasoning | BIG-Bench Hard (3-shot) | 29.1 | 35.0 | 35.3 | 45.2 | 37.1 | 40.4 |
| GSM8K (8-shot) | 31.8 | 46.9 | 46.1 | 61.3 | 49.1 | 76.8 | |
| Average | 40.0 | 49.0 | 50.5 | 50.8 | 55.6 | 60.3 |
Other capabilities
| Capability | Benchmark | MobileLLM-Pro | OLMo 2 1B | SmolLM2 1.7B | Qwen3.5 2B | OLMoE-1B-7B | MobileMoE-L |
|---|---|---|---|---|---|---|---|
| Math | MATH-500 (4-shot) | 8.8 | 10.2 | 15.4 | 31.0 | 8.4 | 39.4 |
| GSM-Plus (5-shot) | 17.0 | 25.1 | 26.6 | 42.4 | 28.1 | 49.7 | |
| Avg | 12.9 | 17.7 | 21.0 | 36.7 | 18.2 | 44.6 | |
| Code | HumanEval | 59.8 | 29.3 | 1.2 | 50.0 | 36.0 | 65.2 |
| MBPP (3-shot) | 44.2 | 14.8 | 34.6 | 41.2 | 30.2 | 54.8 | |
| Avg | 52.0 | 22.0 | 17.9 | 45.6 | 33.1 | 60.0 | |
| Instruction Following | IFEval | 63.1 | 53.5 | 54.7 | 73.3 | 48.1 | 69.7 |
| IFBench | 17.0 | 14.5 | 16.5 | 30.3 | 16.6 | 18.4 | |
| Avg | 40.1 | 34.0 | 35.6 | 51.8 | 32.4 | 44.0 |
Training
MobileMoE uses a four-stage recipe. This checkpoint is the output of stage 3 (supervised fine-tuning).
MobileMoE four-stage training recipe: pre-training (PT) → mid-training (MT) → instruct supervised fine-tuning (SFT) → quantization-aware training (QAT) with INT4 precision.
| Pre-training | Mid-training | SFT | QAT | |
|---|---|---|---|---|
| Context length | 2,048 | 8,192 | 8,192 | 8,192 |
| Total tokens | ~6T | ~500B | ~126B | ~21B |
| Peak learning rate | 4×10-4 | 4×10-5 | 4×10-6 | 4×10-6 |
| LR schedule | Cosine | Linear | Cosine | Cosine |
| Token dispatch | drop-and-pad | drop-and-pad | dropless | dropless |
How to use
MobileMoE uses a custom architecture (model_type: mobilemoe) that is not yet part of upstream transformers, so trust_remote_code=True is required. The modeling code ships in this repo (configuration_mobilemoe.py, modeling_mobilemoe.py).
Requirements
pip install "torch>=2.1" "transformers>=4.57" "safetensors>=0.4" "accelerate>=1.0"
Verified with the following versions:
| Package | Version |
|---|---|
torch |
2.8.0 (cu128) |
transformers |
4.57.6 |
tokenizers |
0.22.2 |
safetensors |
0.7.0 |
accelerate |
1.13.0 |
For batch evaluation we recommend vLLM (≥ 0.10.2) with enforce_eager=True.
Known issues. Loading the tokenizer on transformers 4.57.6 prints a fix_mistral_regex=True warning. Please ignore it and do not set the flag — MobileMoE uses the Llama-3 128k text vocabulary, whose default tokenization is already correct.
Chat
This instruction-tuned model includes a chat template. Format prompts with apply_chat_template; the template uses <|eot|> to mark the end of each turn.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_ID = "facebook/MobileMoE-L-SFT"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
trust_remote_code=True,
dtype=torch.bfloat16,
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()
messages = [{"role": "user", "content": "Why are open-source on-device language models great?"}]
input_ids = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(
input_ids,
attention_mask=torch.ones_like(input_ids),
max_new_tokens=1024,
do_sample=False,
temperature=None,
top_p=None,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))
For multi-turn conversations, append each generated reply to messages with the assistant role. This ensures that each subsequent prompt includes the complete conversation history:
messages = []
for user_message in ["Who are you?", "Why are open-source on-device language models great?"]:
messages.append({"role": "user", "content": user_message})
input_ids = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(
input_ids,
attention_mask=torch.ones_like(input_ids),
max_new_tokens=1024,
do_sample=False,
temperature=None,
top_p=None,
pad_token_id=tokenizer.eos_token_id,
)
reply = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True).strip()
messages.append({"role": "assistant", "content": reply})
print(reply)
Citation
@article{chen2026mobilemoe,
title={MobileMoE: Scaling On-Device Mixture of Experts},
author={Chen, Yanbei and Huang, Hanxian and Chang, Ernie and Szwejbka, Jacob and Desai, Digant and Liu, Zechun and Chandra, Vikas and Krishnamoorthi, Raghuraman},
journal={arXiv preprint arXiv:2605.27358},
year={2026}
}
License
MobileMoE is distributed under the FAIR Noncommercial Research License.