MiMo-V2.5-NVFP4-DFlash

Expert-only NVFP4 quantization of XiaomiMiMo/MiMo-V2.5 (310B MoE, 15B active params) with DFlash speculative decoding for faster inference.

What Was Done

Quantization	Expert-only NVFP4 via NVIDIA ModelOpt
Calibration	64 synthetic calibration samples
Speculative Decoding	DFlash draft model from XiaomiMiMo/MiMo-V2.5-DFlash
KV Cache	FP8
Hardware	Single H200 NVL (141GB VRAM + 252GB RAM)

Quantization Approach

MiMo-V2.5 is a Mixture-of-Experts (MoE) model with 256 routed experts per layer. Only the MoE expert weights are quantized to NVFP4 (4-bit NormalFloat), while attention projections, vision encoder (729M), audio encoder (261M), and other modules remain at higher precision.

This matches Xiaomi's own approach:

"We quantize only the MoE experts to FP4 (MXFP4) and keep the other modules at their original precision." — MiMo-V2.5-Pro-FP4-DFlash

DFlash Speculative Decoding

This model includes Xiaomi's DFlash draft model for speculative decoding:

Draft model	5-layer Qwen3-based model (2.94 GB)
Target layers	[0, 11, 23, 35, 47] (evenly across 48 layers)
Block size	8 tokens
Architecture	Cross-attention to target hidden states

Deployment with DFlash (SGLang)

python3 -m sglang.launch_server \
    --model-path gaber/MiMo-V2.5-NVFP4-DFlash \
    --quantization fp8 \
    --trust-remote-code \
    --dtype bfloat16 \
    --context-length 32768 \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path dflash \
    --speculative-num-draft-tokens 8

Deployment without DFlash (SGLang)

python3 -m sglang.launch_server \
    --model-path gaber/MiMo-V2.5-NVFP4-DFlash \
    --quantization fp8 \
    --trust-remote-code \
    --dtype bfloat16 \
    --context-length 32768

Model Summary

	Original	NVFP4-DFlash
Architecture	MiMoV2ForCausalLM	MiMoV2ForCausalLM
Total Parameters	310B	310B
Active Parameters	15B	15B
Hidden Size	4096	4096
Layers	48 (1 dense + 47 MoE)	48
Routed Experts	256 per layer	256 per layer
Expert Precision	FP8/E8M3	NVFP4
Attention Precision	BF16/FP8	Untouched
Vision Encoder	729M (untouched)	729M (untouched)
Audio Encoder	261M (untouched)	261M (untouched)
Model Size	~295 GB	~137 GB
MTP Layers	3 (329M)	3 (untouched)
DFlash Draft	No	Yes (2.94 GB)

Quantization Config

{
  "quantization_method": "nvfp4_experts",
  "kv_cache_dtype": "fp8",
  "quantization_library": "nvidia-modelopt",
  "calibration_samples": 64,
  "dflash_speculative_decoding": true,
  "dflash_draft_model": "dflash/dflash_draft_model.safetensors"
}

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "gaber/MiMo-V2.5-NVFP4-DFlash",
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(
    "gaber/MiMo-V2.5-NVFP4-DFlash",
    trust_remote_code=True,
)

Hardware Requirements

• Minimum VRAM: ~125 GB (for quantized weights alone)
• Recommended: 2× H100/H200 or 4× A100 80GB
• Single H200 NVL: Works with device_map="auto" (VRAM + RAM split)
• With DFlash: Add ~3 GB VRAM for draft model

Limitations

• MTP Heads: Untouched (3-layer multi-token prediction preserved)
• Vision/Audio: Untouched (729M ViT + 261M Audio encoders preserved)
• DFlash: Requires SGLang with DFlash support for speculative decoding

Base Model

XiaomiMiMo/MiMo-V2.5 — 310B MoE, 15B active, hybrid SWA/GA attention, 1M context, multimodal (text/image/video/audio).

DFlash Source

XiaomiMiMo/MiMo-V2.5-DFlash — DFlash speculative decoding draft model for MiMo-V2.5.

Quantization Tool

NVIDIA ModelOpt — NVFP4 experts-only quantization with 64-sample calibration.

License

Same as base model: MIT