README.md

---
license: fair-noncommercial-research-license
extra_gated_fields:
  First Name: text
  Last Name: text
  Date of birth: date_picker
  Country: country
  Affiliation: text
  Job title:
    type: select
    options:
    - Student
    - Research Graduate
    - AI researcher
    - AI developer/engineer
    - Reporter
    - Other
  geo: ip_location
  By clicking Submit below I accept the terms of the license and acknowledge that the information I provide will be collected stored processed and shared in accordance with the Meta Privacy Policy: checkbox
extra_gated_description: >-
  The information you provide will be collected, stored, processed and shared in
  accordance with the [Meta Privacy
  Policy](https://www.facebook.com/privacy/policy/).
extra_gated_button_content: Submit
language:
- en
library_name: transformers
tags:
- facebook
- meta
- pytorch
- mixture-of-experts
- MoE
- on-device
- quantization
- int4
---

# MobileMoE-L (QAT) Model Card

**MobileMoE** is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion *active* parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a **Base** model (pre-training + mid-training), an **SFT** model (supervised fine-tuning), and a **QAT** model (quantization-aware training). You are currently in the **MobileMoE-L-QAT** repository — the **INT4** quantization-aware-trained 0.9B-active model, built on top of MobileMoE-L-SFT and intended for on-device deployment.

| | **S** | **M** | **L** |
|:---|:---:|:---:|:---:|
| Active / total params | 272M / 1.3B | 528M / 2.8B | 922M / 5.3B |
| Layers | 20 | 26 | 32 |
| Model dimension | 768 | 1024 | 1280 |
| Heads (Q / KV) | 12 / 4 | 16 / 4 | 20 / 4 |
| Routed experts | 60 | 60 | 60 |
| Top-k | 4 | 4 | 4 |
| Projected INT4 weight memory | 0.68 GB | 1.48 GB | **2.75 GB** |
| Base | [MobileMoE-S-Base](https://huggingface.co/facebook/MobileMoE-S-Base) | [MobileMoE-M-Base](https://huggingface.co/facebook/MobileMoE-M-Base) | [MobileMoE-L-Base](https://huggingface.co/facebook/MobileMoE-L-Base) |
| SFT | [MobileMoE-S-SFT](https://huggingface.co/facebook/MobileMoE-S-SFT) | [MobileMoE-M-SFT](https://huggingface.co/facebook/MobileMoE-M-SFT) | [MobileMoE-L-SFT](https://huggingface.co/facebook/MobileMoE-L-SFT) |
| QAT (INT4) | [MobileMoE-S-QAT](https://huggingface.co/facebook/MobileMoE-S-QAT) | [MobileMoE-M-QAT](https://huggingface.co/facebook/MobileMoE-M-QAT) | **MobileMoE-L-QAT** |

For the detailed technical report: 📝 [MobileMoE: Scaling On-Device Mixture of Experts](https://arxiv.org/abs/2605.27358)

For more versions, check out the 🤗 [MobileMoE Collection](https://huggingface.co/collections/facebook/mobilemoe)

![MobileMoE establishes a new Pareto frontier for on-device LLMs](mobilemoe_pareto.png)

**MobileMoE establishes a new Pareto frontier for on-device LLMs.** Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute *F*<sub>inf</sub> = 2*N*<sub>act</sub> (GFLOPs) and (b) total parameters *N*<sub>total</sub> (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB). Accuracy is shown for the instruction-tuned models.

## Key Features

- **A new Pareto frontier for on-device LLMs.** Across 14 foundational benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs at 2–4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters.
- **Scaling-law-derived architecture.** The architecture is derived from an on-device MoE scaling law that jointly optimizes under mobile *memory* and *compute* constraints, identifying an on-device sweet spot: moderate sparsity, with fine-grained experts and shared expert.
- **Four-stage recipe.** Pre-training → mid-training → instruction fine-tuning → INT4 quantization-aware training, all on open-source datasets.

## Model Information

**Model:** MobileMoE-L-QAT (INT4, quantization-aware trained)<br>
**Active Parameters:** 922M<br>
**Total Parameters:** 5.3B<br>
**Layers:** 32<br>
**Model Dimension:** 1280<br>
**Attention Heads:** 20<br>
**KV Heads:** 4 (GQA)<br>
**Head Dimension:** 64<br>
**Routed Experts:** 60 (fine-grained, FFN hidden dim 640 each)<br>
**Active Experts per Token:** 4 (top-k sigmoid routing, with normalization)<br>
**Shared Expert:** 1, always on (FFN hidden dim 2560)<br>
**Vocabulary Size:** 128,256<br>
**Other Features:** QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)<br>
**Input Modality:** Text<br>
**Output Modality:** Text<br>
**Languages:** English<br>
**Chat Template:** Yes (end-of-turn token `<|eot|>`)<br>
**Training Stages:** Pre-training → mid-training → SFT → INT4 QAT<br>
**Context Length:** 8,192 tokens<br>
**Precision:** INT4 weights (symmetric, group size 32; FP16 scales); router stored and routing computed in FP32; BF16 norms and dense compute<br>
**Projected INT4 Weight Memory:** 2.75 GB<br>
**Model Developer:** Meta<br>
**Model Release Date:** Aug 2026<br>
**License:** MobileMoE is FAIR NC licensed

## Quantization

INT4 quantization-aware training is applied to the SFT checkpoint. Weights are quantized as

```
W̃g = sg · clamp(round(Wg / sg), −8, 7),    sg = 2·max(|Wg|) / (2^b − 1),   b = 4
```

where `Wg` is a contiguous group of 32 weights sharing one scale `sg`.

| component | precision |
|:---|:---|
| Attention (q/k/v/o), routed experts, shared expert, embeddings | **INT4**, symmetric, group size 32 |
| Router | Stored and routing computed in FP32 |
| RMSNorm weights, expert bias | BF16 |

**Note:** Projected INT4 weight memory is a weight-only estimate. Packaged checkpoint files (`model.safetensors`) are 0.71 / 1.59 / 3.00 GB for S / M / L, including FP16 group scales, FP32 router weights, and BF16 norms/biases. Runtime memory is backend-dependent and additionally includes the KV cache, activations, and other overhead.

## Results

All numbers below are for **INT4-quantized** models, re-evaluated under identical settings with greedy decoding using [`lm-eval`](https://github.com/EleutherAI/lm-evaluation-harness); few-shot counts are given in parentheses after the benchmark name, and benchmarks shown without one are evaluated 0-shot. All three MobileMoE scales are the exact weights released in their respective QAT repositories, which include brief fine-tuning with self-identity beyond the QAT checkpoints in the [technical report](https://arxiv.org/abs/2605.27358). This produces small differences against the report: averages of 43.9 / 52.7 / 58.1 here versus 44.0 / 52.5 / 57.8 for S / M / L in the report.

### Foundational benchmarks (INT4)

<style>
.mobilemoe-table th{padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#1E3A8A;font-size:14px;line-height:1.2}
.mobilemoe-table th:first-child{width:28%;text-align:left}
.mobilemoe-table th:not(:first-child){width:18%}
.mobilemoe-table td:not([colspan]){padding:7px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:15px;line-height:1.2;vertical-align:middle}
.mobilemoe-table td:first-child:not([colspan]){padding-left:20px;text-align:left}
.mobilemoe-table td:not(:first-child):not([colspan]){text-align:center}
.mobilemoe-table th:nth-child(5),.mobilemoe-table td:nth-child(5):not([colspan]){background:rgba(10,46,254,0.08)}
.mobilemoe-table td[colspan]{padding:8px 12px;text-align:left;font-weight:600;color:#1E3A8A;border-bottom:1px solid rgba(10,46,254,0.2);background:#D6DAFC}
.mobilemoe-table tr.summary-row td{padding:8px 7px;font-weight:600;color:#1E3A8A;border-bottom:1px solid rgba(10,46,254,0.2);background:#D6DAFC!important}
.mobilemoe-table tr.summary-row td:first-child{padding-left:12px}
.mobilemoe-table tr.summary-row td:nth-child(5){background:rgba(10,46,254,0.18)!important}

@media (prefers-color-scheme: dark){
.mobilemoe-table th,.mobilemoe-table td[colspan]{color:#8DA2FF!important;border-bottom-color:#8DA2FF!important}
.mobilemoe-table td[colspan]{background:rgba(10,46,254,0.22)!important}
.mobilemoe-table th:nth-child(5),.mobilemoe-table td:nth-child(5):not([colspan]){background:rgba(10,46,254,0.16)!important}
.mobilemoe-table tr.summary-row td{color:#8DA2FF!important;border-bottom-color:#8DA2FF!important;background:rgba(10,46,254,0.22)!important}
.mobilemoe-table tr.summary-row td:nth-child(5){background:rgba(10,46,254,0.32)!important}
}
</style>

<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1200px;margin:0 auto;padding:16px 0">
<table class="mobilemoe-table" style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
  <thead><tr><th>Benchmark</th><th>MobileLLM-Pro</th><th>MobileMoE-S</th><th>MobileMoE-M</th><th>MobileMoE-L</th></tr></thead>
  <tbody>
    <tr><td><strong>Active / total params</strong></td><td>1.1B</td><td>272M / 1.3B</td><td>528M / 2.8B</td><td>922M / 5.3B</td></tr>
    <tr><td colspan="5">Commonsense Reasoning</td></tr>
    <tr><td><strong>HellaSwag</strong></td><td>64.7</td><td>53.3</td><td>63.7</td><td>70.7</td></tr>
    <tr><td><strong>PIQA</strong></td><td>75.6</td><td>73.9</td><td>76.4</td><td>78.9</td></tr>
    <tr><td><strong>SIQA</strong></td><td>47.4</td><td>45.3</td><td>48.7</td><td>53.4</td></tr>
    <tr><td><strong>WinoGrande</strong></td><td>62.8</td><td>56.6</td><td>61.6</td><td>65.4</td></tr>
    <tr><td colspan="5">Knowledge</td></tr>
    <tr><td><strong>MMLU (5-shot)</strong></td><td>30.4</td><td>39.0</td><td>52.2</td><td>57.4</td></tr>
    <tr><td><strong>NaturalQuestions (5-shot)</strong></td><td>13.9</td><td>8.6</td><td>4.7</td><td>19.3</td></tr>
    <tr><td><strong>TriviaQA (5-shot)</strong></td><td>39.9</td><td>25.2</td><td>42.7</td><td>52.7</td></tr>
    <tr><td colspan="5">Science</td></tr>
    <tr><td><strong>ARC-Challenge (25-shot)</strong></td><td>51.6</td><td>42.0</td><td>51.5</td><td>55.6</td></tr>
    <tr><td><strong>ARC-Easy</strong></td><td>75.2</td><td>69.3</td><td>79.2</td><td>80.6</td></tr>
    <tr><td><strong>OpenBookQA</strong></td><td>42.8</td><td>30.6</td><td>36.6</td><td>40.4</td></tr>
    <tr><td colspan="5">Reading</td></tr>
    <tr><td><strong>BoolQ</strong></td><td>76.8</td><td>71.2</td><td>77.1</td><td>78.3</td></tr>
    <tr><td><strong>DROP (3-shot)</strong></td><td>20.5</td><td>26.3</td><td>43.5</td><td>47.3</td></tr>
    <tr><td colspan="5">Reasoning</td></tr>
    <tr><td><strong>BIG-Bench Hard (3-shot)</strong></td><td>31.4</td><td>30.8</td><td>37.0</td><td>39.5</td></tr>
    <tr><td><strong>GSM8K (8-shot)</strong></td><td>4.1</td><td>42.9</td><td>63.5</td><td>73.2</td></tr>
    <tr class="summary-row"><td><strong>Average</strong></td><td>45.5</td><td>43.9</td><td>52.7</td><td>58.1</td></tr>
  </tbody>
</table>
</div>

## Training

MobileMoE uses a four-stage recipe. This checkpoint is the output of stage 4 (INT4 QAT).

![MobileMoE four-stage training recipe](mobilemoe_recipe.png)

**MobileMoE four-stage training recipe:** pre-training (PT) → mid-training (MT) → instruct supervised fine-tuning (SFT) → quantization-aware training (QAT) with INT4 precision.

| | Pre-training | Mid-training | SFT | **QAT** |
|---|---|---|---|---|
| Context length | 2,048 | 8,192 | 8,192 | **8,192** |
| Total tokens | ~6T | ~500B | ~126B | **~21B** |
| Peak learning rate | 4×10<sup>-4</sup> | 4×10<sup>-5</sup> | 4×10<sup>-6</sup> | **4×10<sup>-6</sup>** |
| LR schedule | Cosine | Linear | Cosine | **Cosine** |
| Token dispatch | drop-and-pad | drop-and-pad | dropless | **dropless** |

## How to use

MobileMoE uses a custom architecture (`model_type: mobilemoe`) that is not yet part of upstream `transformers`, so **`trust_remote_code=True` is required**. The modeling code ships in this repo (`configuration_mobilemoe.py`, `modeling_mobilemoe.py`), and it handles INT4 dequantization automatically.

### Requirements

```bash
pip install "torch>=2.1" "transformers>=4.57" "safetensors>=0.4" "accelerate>=1.0"
```

Verified with the following versions:

| Package | Version |
|---|---|
| `torch` | 2.8.0 (cu128) |
| `transformers` | 4.57.6 |
| `tokenizers` | 0.22.2 |
| `safetensors` | 0.7.0 |
| `accelerate` | 1.13.0 |

### Chat

This model is **instruction-tuned** (QAT is applied on top of the SFT checkpoint), so use the chat template:

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "facebook/MobileMoE-L-QAT"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

messages = [{"role": "user", "content": "Why are open-source on-device language models great?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_new_tokens=1024,
    do_sample=False,
    temperature=None,
    top_p=None,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

For multi-turn conversations, append each generated reply to `messages` with the `assistant` role. This ensures that each subsequent prompt includes the complete conversation history:

```python
messages = []

for user_message in ["Who are you?", "Why are open-source on-device language models great?"]:
    messages.append({"role": "user", "content": user_message})
    input_ids = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    outputs = model.generate(
        input_ids,
        attention_mask=torch.ones_like(input_ids),
        max_new_tokens=1024,
        do_sample=False,
        temperature=None,
        top_p=None,
        pad_token_id=tokenizer.eos_token_id,
    )
    reply = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True).strip()
    messages.append({"role": "assistant", "content": reply})
    print(reply)
```

**Known issues.**

- Loading the tokenizer on `transformers` 4.57.6 prints a `fix_mistral_regex=True` warning. Please ignore it and do not set the flag, as MobileMoE uses the Llama-3 tokenizer whose default tokenization is already correct.
- The checkpoint stores `qweight` / `weight_scale` and creates dense weights on the first forward pass, so a *"newly initialized"* warning before then is expected.
- vLLM does not support this repository's INT4 format directly because its Transformers fallback replaces the custom layers and ignores the INT4 tensors. Convert the checkpoint to BF16 as described below, or use [MobileMoE-L-SFT](https://huggingface.co/facebook/MobileMoE-L-SFT) instead.

### Exporting the INT4 Checkpoint to BF16

No conversion is needed for general use (e.g., with `transformers`) because the custom modeling code dequantizes the INT4 checkpoint to BF16 on the first forward pass. This section exports a numerically identical BF16 checkpoint to support other runtimes (e.g., vLLM) that bypass the custom loading logic and cannot use `qweight` / `weight_scale` directly.

```python
import os, shutil, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

SRC, DST = "facebook/MobileMoE-L-QAT", "./MobileMoE-L-QAT-bf16"

model = AutoModelForCausalLM.from_pretrained(SRC, trust_remote_code=True, dtype=torch.bfloat16)
model(torch.tensor([[128000]]))                 # first forward materializes INT4 -> BF16

state = {k: v for k, v in model.state_dict().items()
         if not (k.endswith("qweight") or k.endswith("_scale"))}
model.config.quantization = None                # mark the copy as plain BF16
model.save_pretrained(DST, state_dict=state, safe_serialization=True)
AutoTokenizer.from_pretrained(SRC).save_pretrained(DST)

# a trust_remote_code architecture needs its modeling files alongside the weights
for f in ("modeling_mobilemoe.py", "configuration_mobilemoe.py"):
    src_f = os.path.join(SRC, f) if os.path.isdir(SRC) else __import__(
        "huggingface_hub").hf_hub_download(SRC, f)
    shutil.copy2(src_f, os.path.join(DST, f))
```

## Citation

```bibtex
@article{chen2026mobilemoe,
  title={MobileMoE: Scaling On-Device Mixture of Experts},
  author={Chen, Yanbei and Huang, Hanxian and Chang, Ernie and Szwejbka, Jacob and Desai, Digant and Liu, Zechun and Chandra, Vikas and Krishnamoorthi, Raghuraman},
  journal={arXiv preprint arXiv:2605.27358},
  year={2026}
}
```

## License

MobileMoE is distributed under the [FAIR Noncommercial Research License](https://huggingface.co/facebook/MobileMoE-L-QAT/blob/main/LICENSE).