README.md

---
license: apache-2.0
language:
- zh
- yue
- en
pipeline_tag: automatic-speech-recognition
library_name: transformers
tags:
- asr
- speech-recognition
- chinese
- dialect
- qwen3-asr
- audio
base_model: Qwen/Qwen3-ASR-1.7B
base_model_relation: finetune
---

<p align="center">
  <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/logo.jpeg" width="520" alt="CN-MultiDialect-ASR logo">
</p>


**On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin**

<div align="center">
  <p><strong>Shuiyuan Wang<sup>1</sup> · Bingshen Mu<sup>1</sup> · Pengshen Zhang<sup>2</sup> · Chengyou Wang<sup>1</sup> · Yujie Liao<sup>1</sup> · Chengdong Liang<sup>2</sup> · Binbin Zhang<sup>2</sup> · Qiangze Feng<sup>3</sup> · Lei Xie<sup>1</sup></strong></p>
  <p><sup>1</sup> Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University, Xi'an, China<br>
  <sup>2</sup> WeNet Community<br>
  <sup>3</sup> NEXDATA TECHNOLOGY INC.</p>
</div>

<div align="center">

[![Paper](https://img.shields.io/badge/Paper-arXiv-blue)](https://arxiv.org/abs/2608.11898)
[![GitHub](https://img.shields.io/badge/GitHub-ASLP--lab%2FCN--MultiDialect--ASR-black)](https://github.com/ASLP-lab/CN-MultiDialect-ASR)
[![License](https://img.shields.io/badge/License-Apache%202.0-green)](https://huggingface.co/ASLP-lab/CN-MultiDialect-ASR)

</div>

This repository hosts the released **CN-MultiDialect-ASR** checkpoint, adapted from [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) with a three-stage pipeline: continual pre-training (CPT), dialect supervised fine-tuning (SFT), and On-Policy Self-Distillation (OPSD). The goal is to improve Chinese dialect recognition **without raising Mandarin CER**.

- Paper: [arXiv:2608.11898](https://arxiv.org/abs/2608.11898)
- Code, demo, and training scripts: [ASLP-lab/CN-MultiDialect-ASR](https://github.com/ASLP-lab/CN-MultiDialect-ASR)

<div align="center">
  <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/opsd.png" alt="OPSD framework" width="90%">
  <p><em>Overview of the staged adaptation pipeline. Top: base model, CPT, SFT, and OPSD. Bottom: OPSD with student on-policy prefixes, a frozen teacher conditioned on the reference transcript as privileged context, soft targets q<sub>t</sub>, and token-level KL.</em></p>
</div>

## Demo

Video demo with live waveforms and model transcriptions for Mandarin, English, four core dialects, and 15 ChinaVoices dialects.

<video src="https://github.com/user-attachments/assets/29439247-bc62-45e1-8119-e0c45416c957" controls preload="metadata" playsinline width="100%" aria-label="CN-MultiDialect-ASR video demo"></video>

## Key Features

- **Mandarin–dialect balanced adaptation**: improves Chinese dialect ASR while retaining Mandarin recognition.
- **Three-stage pipeline**: CPT strengthens the Chinese ASR foundation, dialect SFT specializes for dialects, and OPSD refines the final checkpoint.
- **On-Policy Self-Distillation**: trains on student-decoded prefixes with soft teacher targets, reducing the train–test mismatch of teacher-forced ASR training.
- **Drop-in inference**: compatible with the official [`qwen-asr`](https://github.com/QwenLM/Qwen3-ASR) package.

## Quickstart

Inference is compatible with [Qwen3-ASR](https://github.com/QwenLM/Qwen3-ASR). We recommend installing the official `qwen-asr` package in a clean environment.

### Environment Setup

```bash
conda create -n qwen3-asr python=3.12 -y
conda activate qwen3-asr
pip install -U qwen-asr
```

For faster inference with the vLLM backend:

```bash
pip install -U qwen-asr[vllm]
```

### Model Download

You can load the model directly from Hugging Face, or download it locally first:

```bash
# Hugging Face
pip install -U "huggingface_hub[cli]"
hf download ASLP-lab/CN-MultiDialect-ASR --local-dir ./CN-MultiDialect-ASR

# ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model ASLP-lab/CN-MultiDialect-ASR --local_dir ./CN-MultiDialect-ASR
```

### Python Inference

Load the model with `Qwen3ASRModel.from_pretrained` and call `transcribe`:

```python
import torch
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "ASLP-lab/CN-MultiDialect-ASR",  # or "./CN-MultiDialect-ASR" for a local path
    dtype=torch.bfloat16,
    device_map="cuda:0",
    # attn_implementation="flash_attention_2",
    max_inference_batch_size=32,
    max_new_tokens=256,
)

results = model.transcribe(
    audio="path/to/audio.wav",
    language="Chinese",  # or None for automatic language detection
)

print(results[0].language)
print(results[0].text)
```

Batch inference is also supported:

```python
results = model.transcribe(
    audio=[
        "path/to/mandarin.wav",
        "path/to/dialect.wav",
    ],
    language=["Chinese", "Chinese"],
)

for r in results:
    print(r.language, r.text)
```

For vLLM backend, streaming inference, and forced alignment, see the [Qwen3-ASR repository](https://github.com/QwenLM/Qwen3-ASR).

## Method Overview

| Stage | Training data | Goal | Objective |
|:-----:|:--------------|:-----|:----------|
| `CPT` | Full Mandarin-dialect collection (`~100k` hours) | Build a stronger Chinese ASR foundation | Cross-entropy |
| `SFT` | Same sources with higher dialect sampling weight and a small Mandarin anchor | Lower dialect CER | Cross-entropy |
| `OPSD` | Dialect refinement subset (`~5k` hours) | Improve dialect recognition without hurting Mandarin | Token-level KL |

At inference time, only the student pathway is used.

## Performances

### Dialect Overview

<div align="center">
  <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/radar_1_cer_panels.png" alt="Side-by-side radar of 1-CER on public and internal dialect sets" width="92%">
  <p><em>Higher is better. Left: 5 public dialect sets; right: 18 internal dialects. Both panels use the same radial scale (0.2–1.0). The figure compares the Qwen3-ASR baseline with the released <strong>CN-MultiDialect-ASR</strong> (OPSD) checkpoint.</em></p>
</div>

### Public Dialect CER (%)

<table>
  <thead>
    <tr>
      <th align="center">Evaluation set</th>
      <th align="center">Dialect</th>
      <th align="center">Qwen3-ASR</th>
      <th align="center">CN-MultiDialect-ASR</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td align="center">WenetSpeech-Yue Long</td>
      <td align="center">Cantonese</td>
      <td align="center">9.99</td>
      <td align="center"><b>8.80</b></td>
    </tr>
    <tr>
      <td align="center">WenetSpeech-Yue Short</td>
      <td align="center">Cantonese</td>
      <td align="center">6.93</td>
      <td align="center"><b>5.31</b></td>
    </tr>
    <tr>
      <td align="center">WenetSpeech-Chuan Easy</td>
      <td align="center">Sichuan</td>
      <td align="center">12.38</td>
      <td align="center"><b>11.86</b></td>
    </tr>
    <tr>
      <td align="center">WenetSpeech-Chuan Hard</td>
      <td align="center">Sichuan</td>
      <td align="center">21.79</td>
      <td align="center"><b>21.74</b></td>
    </tr>
    <tr>
      <td align="center">WenetSpeech-Wu</td>
      <td align="center">Wu</td>
      <td align="center">25.74</td>
      <td align="center"><b>16.26</b></td>
    </tr>
    <tr>
      <td align="center"><b>Dialect Avg.</b></td>
      <td align="center"></td>
      <td align="center">15.37</td>
      <td align="center"><b>12.79</b></td>
    </tr>
  </tbody>
</table>

### Internal Dialect CER (%)

<table>
  <thead>
    <tr>
      <th align="center">Dialect</th>
      <th align="center">Qwen3-ASR</th>
      <th align="center">CN-MultiDialect-ASR</th>
    </tr>
  </thead>
  <tbody>
    <tr><td align="center">Anhui</td><td align="center">18.95</td><td align="center"><b>13.08</b></td></tr>
    <tr><td align="center">Cantonese</td><td align="center">10.06</td><td align="center"><b>7.74</b></td></tr>
    <tr><td align="center">Changsha</td><td align="center">14.79</td><td align="center"><b>10.23</b></td></tr>
    <tr><td align="center">Chaoshan</td><td align="center">45.59</td><td align="center"><b>25.21</b></td></tr>
    <tr><td align="center">Dongbei</td><td align="center">6.45</td><td align="center"><b>5.80</b></td></tr>
    <tr><td align="center">Henan</td><td align="center">8.46</td><td align="center"><b>5.99</b></td></tr>
    <tr><td align="center">Kejia</td><td align="center">60.47</td><td align="center"><b>28.60</b></td></tr>
    <tr><td align="center">Minnan</td><td align="center">30.03</td><td align="center"><b>18.59</b></td></tr>
    <tr><td align="center">Nanchang</td><td align="center">33.41</td><td align="center"><b>15.58</b></td></tr>
    <tr><td align="center">Nanjing</td><td align="center">13.37</td><td align="center"><b>9.33</b></td></tr>
    <tr><td align="center">Shanxi</td><td align="center">28.53</td><td align="center"><b>18.69</b></td></tr>
    <tr><td align="center">Shaanxi</td><td align="center">9.68</td><td align="center"><b>6.28</b></td></tr>
    <tr><td align="center">Shandong</td><td align="center">8.78</td><td align="center"><b>7.64</b></td></tr>
    <tr><td align="center">Shanghai</td><td align="center">15.78</td><td align="center">12.07</td></tr>
    <tr><td align="center">Sichuan</td><td align="center">5.99</td><td align="center">5.38</td></tr>
    <tr><td align="center">Suzhou</td><td align="center">50.35</td><td align="center"><b>20.73</b></td></tr>
    <tr><td align="center">Wuhan</td><td align="center">11.30</td><td align="center"><b>7.59</b></td></tr>
    <tr><td align="center">Xuzhou</td><td align="center">6.12</td><td align="center"><b>5.04</b></td></tr>
    <tr><td align="center"><b>Internal Avg.</b></td><td align="center">21.01</td><td align="center"><b>12.42</b></td></tr>
  </tbody>
</table>

### Mandarin CER (%)

<table>
  <thead>
    <tr>
      <th align="center">Evaluation set</th>
      <th align="center">Qwen3-ASR</th>
      <th align="center">CN-MultiDialect-ASR</th>
    </tr>
  </thead>
  <tbody>
    <tr><td align="center">AISHELL-1</td><td align="center">1.57</td><td align="center"><b>1.38</b></td></tr>
    <tr><td align="center">AISHELL-2</td><td align="center">2.79</td><td align="center"><b>2.52</b></td></tr>
    <tr><td align="center">KeSpeech</td><td align="center">5.11</td><td align="center"><b>4.56</b></td></tr>
    <tr><td align="center">SpeechIO-1</td><td align="center"><b>0.75</b></td><td align="center">0.86</td></tr>
    <tr><td align="center">SpeechIO-2</td><td align="center">3.83</td><td align="center"><b>3.39</b></td></tr>
    <tr><td align="center">SpeechIO-3</td><td align="center">1.39</td><td align="center"><b>1.27</b></td></tr>
    <tr><td align="center">Test_Meeting</td><td align="center"><b>6.74</b></td><td align="center">6.85</td></tr>
    <tr><td align="center">Test_Net</td><td align="center">5.46</td><td align="center"><b>5.30</b></td></tr>
    <tr><td align="center"><b>Mandarin Avg.</b></td><td align="center">3.46</td><td align="center"><b>3.27</b></td></tr>
  </tbody>
</table>

## Citation

If you use this model, please cite:

```bibtex
@misc{wang2026onpolicyselfdistillationmultidialectasr,
  title={On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin},
  author={Shuiyuan Wang and Bingshen Mu and Pengshen Zhang and Chengyou Wang and Yujie Liao and Chengdong Liang and Binbin Zhang and Qiangze Feng and Lei Xie},
  year={2026},
  eprint={2608.11898},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  url={https://arxiv.org/abs/2608.11898}
}
```

## License

The released model is licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).

## Contact

For questions or collaborations, please contact [wangshuiyuan@mail.nwpu.edu.cn](mailto:wangshuiyuan@mail.nwpu.edu.cn).

You are also welcome to join our WeChat group for technical discussions and updates.

<p align="center">
  <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/wechat.jpg" width="300" alt="WeChat group QR code">
  <br>
  <em>Scan to join our WeChat discussion group</em>
</p>