---
license: mit
language:
- zh
- en
- yue
pipeline_tag: automatic-speech-recognition
tags:
- safetensors
- fp8
- quantization
- speech-recognition
base_model: XiaomiMiMo/MiMo-V2.5-ASR
---

# MiMo-V2.5-ASR — FP8 (e4m3fn)

FP8-quantized build of [XiaomiMiMo/MiMo-V2.5-ASR](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR),
the Xiaomi MiMo end-to-end ASR model with native Mandarin/English code-switching,
Chinese dialects, song lyrics, noisy/multi-speaker robustness, and native punctuation.

## What this is

- **Weights:** `float8_e4m3fn`, per-output-channel absmax scaling (one fp32 scale per row), baked at save time.
- **Activations:** dynamic per-tensor fp8 quantization each forward pass.
- **Matmul:** `torch._scaled_mm` (FP8 tensor cores on Ada / Hopper / Blackwell).
- **Skipped (kept bf16):** embeddings, RMSNorm/LayerNorm, biases. 417 Linear layers converted.
- The audio encoder/tokenizer ([MiMo-Audio-Tokenizer](https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer)) is **not** quantized; download it separately for inference.

This roughly halves the LLM weight footprint (~32 GB bf16 → ~16-17 GB on disk).

## Important: this is NOT a drop-in `from_pretrained` checkpoint

`model.safetensors` stores custom `FP8Linear` buffers (`*.weight_fp8`, `*.weight_scale`),
not standard HF Linear weights. It must be loaded through the matching `FP8Linear`
modules. Use the loader below.

## Usage

```bash
git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git
cd MiMo-V2.5-ASR
pip install -r requirements.txt
pip install flash-attn==2.7.4.post1   # required by the audio tokenizer
hf download XiaomiMiMo/MiMo-Audio-Tokenizer --local-dir ./models/MiMo-Audio-Tokenizer
hf download Infatoshi/MiMo-V2.5-ASR-FP8 --local-dir ./MiMo-V2.5-ASR-FP8
```

Then load with the `FP8Linear` loader (`quantize_fp8.py`, included here as `quantize_fp8.py`):

```python
from quantize_fp8 import load_fp8_model
mimo = load_fp8_model(
    fp8_dir="./MiMo-V2.5-ASR-FP8",
    tokenizer_path="./models/MiMo-Audio-Tokenizer",
    repo_root=".",                     # the cloned MiMo-V2.5-ASR repo
)
print(mimo.asr_sft("audio.wav", audio_tag="<english>"))
```

## Quantization fidelity

Per-output-channel absmax dequant error vs the original fp32 weights, sampled across
depth (layers 0/17/35), all attn+mlp projections, lm_head, and the audio local transformer:

- relative Frobenius error: **~0.026, uniform** across every sampled layer (max 0.027 on lm_head)
- no corrupted or outlier layers

This is the expected magnitude for fp8 e4m3 with per-channel scaling (3 mantissa bits).

## Requirements

- CUDA GPU with FP8 tensor cores (Ada / Hopper / Blackwell), CUDA >= 12.0
- torch >= 2.6, safetensors
- **Blackwell (sm_120, e.g. RTX PRO 6000 / RTX 50xx):** use a torch build with CUDA 12.8+
  (torch >= 2.7, `cu128`). torch 2.6 `cu124` ships no sm_120 kernels and will fail with
  "no kernel image is available for execution on the device".

## Notes / caveats

- FP8 e4m3fn weight-only-style quantization is lossy; expect small WER deltas vs bf16.
- Per-tensor dynamic activation scaling is simple and fast but less accurate than
  per-token scaling on activations with large outliers.

Derivative of an MIT-licensed model; original credit to the Xiaomi MiMo team.