Hugging Face's logo

ASLP-lab
/
CN-MultiDialect-ASR

Automatic Speech Recognition
Transformers
Safetensors
Chinese
Yue Chinese
English
qwen3_asr
asr
speech-recognition
chinese
dialect
qwen3-asr
audio
Model card Files
xet
 Community
CN-MultiDialect-ASR / README.md
ASLP-lab's picture
ASLP-lab
Update README.md
bea6351 verified
preview code
|
Raw
History Blame Contribute Delete
12.1 kB
---
license: apache-2.0
language:
- zh
- yue
- en
pipeline_tag: automatic-speech-recognition
library_name: transformers
tags:
- asr
- speech-recognition
- chinese
- dialect
- qwen3-asr
- audio
base_model: Qwen/Qwen3-ASR-1.7B
base_model_relation: finetune
---
<p align="center">
 <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/logo.jpeg" width="520" alt="CN-MultiDialect-ASR logo">
</p>
**On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin**
<div align="center">
 <p><strong>Shuiyuan Wang<sup>1</sup> · Bingshen Mu<sup>1</sup> · Pengshen Zhang<sup>2</sup> · Chengyou Wang<sup>1</sup> · Yujie Liao<sup>1</sup> · Chengdong Liang<sup>2</sup> · Binbin Zhang<sup>2</sup> · Qiangze Feng<sup>3</sup> · Lei Xie<sup>1</sup></strong></p>
 <p><sup>1</sup> Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University, Xi'an, China<br>
 <sup>2</sup> WeNet Community<br>
 <sup>3</sup> NEXDATA TECHNOLOGY INC.</p>
</div>
<div align="center">
[![Paper](https://img.shields.io/badge/Paper-arXiv-blue)](https://arxiv.org/abs/2608.11898)
[![GitHub](https://img.shields.io/badge/GitHub-ASLP--lab%2FCN--MultiDialect--ASR-black)](https://github.com/ASLP-lab/CN-MultiDialect-ASR)
[![License](https://img.shields.io/badge/License-Apache%202.0-green)](https://huggingface.co/ASLP-lab/CN-MultiDialect-ASR)
</div>
This repository hosts the released **CN-MultiDialect-ASR** checkpoint, adapted from [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) with a three-stage pipeline: continual pre-training (CPT), dialect supervised fine-tuning (SFT), and On-Policy Self-Distillation (OPSD). The goal is to improve Chinese dialect recognition **without raising Mandarin CER**.
- Paper: [arXiv:2608.11898](https://arxiv.org/abs/2608.11898)
- Code, demo, and training scripts: [ASLP-lab/CN-MultiDialect-ASR](https://github.com/ASLP-lab/CN-MultiDialect-ASR)
<div align="center">
 <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/opsd.png" alt="OPSD framework" width="90%">
 <p><em>Overview of the staged adaptation pipeline. Top: base model, CPT, SFT, and OPSD. Bottom: OPSD with student on-policy prefixes, a frozen teacher conditioned on the reference transcript as privileged context, soft targets q<sub>t</sub>, and token-level KL.</em></p>
</div>
## Demo
Video demo with live waveforms and model transcriptions for Mandarin, English, four core dialects, and 15 ChinaVoices dialects.
<video src="https://github.com/user-attachments/assets/29439247-bc62-45e1-8119-e0c45416c957" controls preload="metadata" playsinline width="100%" aria-label="CN-MultiDialect-ASR video demo"></video>
## Key Features
- **Mandarin–dialect balanced adaptation**: improves Chinese dialect ASR while retaining Mandarin recognition.
- **Three-stage pipeline**: CPT strengthens the Chinese ASR foundation, dialect SFT specializes for dialects, and OPSD refines the final checkpoint.
- **On-Policy Self-Distillation**: trains on student-decoded prefixes with soft teacher targets, reducing the train–test mismatch of teacher-forced ASR training.
- **Drop-in inference**: compatible with the official [`qwen-asr`](https://github.com/QwenLM/Qwen3-ASR) package.
## Quickstart
Inference is compatible with [Qwen3-ASR](https://github.com/QwenLM/Qwen3-ASR). We recommend installing the official `qwen-asr` package in a clean environment.
### Environment Setup
```bash
conda create -n qwen3-asr python=3.12 -y
conda activate qwen3-asr
pip install -U qwen-asr
```
For faster inference with the vLLM backend:
```bash
pip install -U qwen-asr[vllm]
```
### Model Download
You can load the model directly from Hugging Face, or download it locally first:
```bash
# Hugging Face
pip install -U "huggingface_hub[cli]"
hf download ASLP-lab/CN-MultiDialect-ASR --local-dir ./CN-MultiDialect-ASR
# ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model ASLP-lab/CN-MultiDialect-ASR --local_dir ./CN-MultiDialect-ASR
```
### Python Inference
Load the model with `Qwen3ASRModel.from_pretrained` and call `transcribe`:
```python
import torch
from qwen_asr import Qwen3ASRModel
model = Qwen3ASRModel.from_pretrained(
 "ASLP-lab/CN-MultiDialect-ASR", # or "./CN-MultiDialect-ASR" for a local path
 dtype=torch.bfloat16,
 device_map="cuda:0",
 # attn_implementation="flash_attention_2",
 max_inference_batch_size=32,
 max_new_tokens=256,
)
results = model.transcribe(
 audio="path/to/audio.wav",
 language="Chinese", # or None for automatic language detection
)
print(results[0].language)
print(results[0].text)
```
Batch inference is also supported:
```python
results = model.transcribe(
 audio=[
 "path/to/mandarin.wav",
 "path/to/dialect.wav",
 ],
 language=["Chinese", "Chinese"],
)
for r in results:
 print(r.language, r.text)
```
For vLLM backend, streaming inference, and forced alignment, see the [Qwen3-ASR repository](https://github.com/QwenLM/Qwen3-ASR).
## Method Overview
| Stage | Training data | Goal | Objective |
|:-----:|:--------------|:-----|:----------|
| `CPT` | Full Mandarin-dialect collection (`~100k` hours) | Build a stronger Chinese ASR foundation | Cross-entropy |
| `SFT` | Same sources with higher dialect sampling weight and a small Mandarin anchor | Lower dialect CER | Cross-entropy |
| `OPSD` | Dialect refinement subset (`~5k` hours) | Improve dialect recognition without hurting Mandarin | Token-level KL |
At inference time, only the student pathway is used.
## Performances
### Dialect Overview
<div align="center">
 <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/radar_1_cer_panels.png" alt="Side-by-side radar of 1-CER on public and internal dialect sets" width="92%">
 <p><em>Higher is better. Left: 5 public dialect sets; right: 18 internal dialects. Both panels use the same radial scale (0.2–1.0). The figure compares the Qwen3-ASR baseline with the released <strong>CN-MultiDialect-ASR</strong> (OPSD) checkpoint.</em></p>
</div>
### Public Dialect CER (%)
<table>
 <thead>
 <tr>
 <th align="center">Evaluation set</th>
 <th align="center">Dialect</th>
 <th align="center">Qwen3-ASR</th>
 <th align="center">CN-MultiDialect-ASR</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td align="center">WenetSpeech-Yue Long</td>
 <td align="center">Cantonese</td>
 <td align="center">9.99</td>
 <td align="center"><b>8.80</b></td>
 </tr>
 <tr>
 <td align="center">WenetSpeech-Yue Short</td>
 <td align="center">Cantonese</td>
 <td align="center">6.93</td>
 <td align="center"><b>5.31</b></td>
 </tr>
 <tr>
 <td align="center">WenetSpeech-Chuan Easy</td>
 <td align="center">Sichuan</td>
 <td align="center">12.38</td>
 <td align="center"><b>11.86</b></td>
 </tr>
 <tr>
 <td align="center">WenetSpeech-Chuan Hard</td>
 <td align="center">Sichuan</td>
 <td align="center">21.79</td>
 <td align="center"><b>21.74</b></td>
 </tr>
 <tr>
 <td align="center">WenetSpeech-Wu</td>
 <td align="center">Wu</td>
 <td align="center">25.74</td>
 <td align="center"><b>16.26</b></td>
 </tr>
 <tr>
 <td align="center"><b>Dialect Avg.</b></td>
 <td align="center"></td>
 <td align="center">15.37</td>
 <td align="center"><b>12.79</b></td>
 </tr>
 </tbody>
</table>
### Internal Dialect CER (%)
<table>
 <thead>
 <tr>
 <th align="center">Dialect</th>
 <th align="center">Qwen3-ASR</th>
 <th align="center">CN-MultiDialect-ASR</th>
 </tr>
 </thead>
 <tbody>
 <tr><td align="center">Anhui</td><td align="center">18.95</td><td align="center"><b>13.08</b></td></tr>
 <tr><td align="center">Cantonese</td><td align="center">10.06</td><td align="center"><b>7.74</b></td></tr>
 <tr><td align="center">Changsha</td><td align="center">14.79</td><td align="center"><b>10.23</b></td></tr>
 <tr><td align="center">Chaoshan</td><td align="center">45.59</td><td align="center"><b>25.21</b></td></tr>
 <tr><td align="center">Dongbei</td><td align="center">6.45</td><td align="center"><b>5.80</b></td></tr>
 <tr><td align="center">Henan</td><td align="center">8.46</td><td align="center"><b>5.99</b></td></tr>
 <tr><td align="center">Kejia</td><td align="center">60.47</td><td align="center"><b>28.60</b></td></tr>
 <tr><td align="center">Minnan</td><td align="center">30.03</td><td align="center"><b>18.59</b></td></tr>
 <tr><td align="center">Nanchang</td><td align="center">33.41</td><td align="center"><b>15.58</b></td></tr>
 <tr><td align="center">Nanjing</td><td align="center">13.37</td><td align="center"><b>9.33</b></td></tr>
 <tr><td align="center">Shanxi</td><td align="center">28.53</td><td align="center"><b>18.69</b></td></tr>
 <tr><td align="center">Shaanxi</td><td align="center">9.68</td><td align="center"><b>6.28</b></td></tr>
 <tr><td align="center">Shandong</td><td align="center">8.78</td><td align="center"><b>7.64</b></td></tr>
 <tr><td align="center">Shanghai</td><td align="center">15.78</td><td align="center">12.07</td></tr>
 <tr><td align="center">Sichuan</td><td align="center">5.99</td><td align="center">5.38</td></tr>
 <tr><td align="center">Suzhou</td><td align="center">50.35</td><td align="center"><b>20.73</b></td></tr>
 <tr><td align="center">Wuhan</td><td align="center">11.30</td><td align="center"><b>7.59</b></td></tr>
 <tr><td align="center">Xuzhou</td><td align="center">6.12</td><td align="center"><b>5.04</b></td></tr>
 <tr><td align="center"><b>Internal Avg.</b></td><td align="center">21.01</td><td align="center"><b>12.42</b></td></tr>
 </tbody>
</table>
### Mandarin CER (%)
<table>
 <thead>
 <tr>
 <th align="center">Evaluation set</th>
 <th align="center">Qwen3-ASR</th>
 <th align="center">CN-MultiDialect-ASR</th>
 </tr>
 </thead>
 <tbody>
 <tr><td align="center">AISHELL-1</td><td align="center">1.57</td><td align="center"><b>1.38</b></td></tr>
 <tr><td align="center">AISHELL-2</td><td align="center">2.79</td><td align="center"><b>2.52</b></td></tr>
 <tr><td align="center">KeSpeech</td><td align="center">5.11</td><td align="center"><b>4.56</b></td></tr>
 <tr><td align="center">SpeechIO-1</td><td align="center"><b>0.75</b></td><td align="center">0.86</td></tr>
 <tr><td align="center">SpeechIO-2</td><td align="center">3.83</td><td align="center"><b>3.39</b></td></tr>
 <tr><td align="center">SpeechIO-3</td><td align="center">1.39</td><td align="center"><b>1.27</b></td></tr>
 <tr><td align="center">Test_Meeting</td><td align="center"><b>6.74</b></td><td align="center">6.85</td></tr>
 <tr><td align="center">Test_Net</td><td align="center">5.46</td><td align="center"><b>5.30</b></td></tr>
 <tr><td align="center"><b>Mandarin Avg.</b></td><td align="center">3.46</td><td align="center"><b>3.27</b></td></tr>
 </tbody>
</table>
## Citation
If you use this model, please cite:
```bibtex
@misc{wang2026onpolicyselfdistillationmultidialectasr,
 title={On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin},
 author={Shuiyuan Wang and Bingshen Mu and Pengshen Zhang and Chengyou Wang and Yujie Liao and Chengdong Liang and Binbin Zhang and Qiangze Feng and Lei Xie},
 year={2026},
 eprint={2608.11898},
 archivePrefix={arXiv},
 primaryClass={eess.AS},
 url={https://arxiv.org/abs/2608.11898}
}
```
## License
The released model is licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).
## Contact
For questions or collaborations, please contact [wangshuiyuan@mail.nwpu.edu.cn](mailto:wangshuiyuan@mail.nwpu.edu.cn).
You are also welcome to join our WeChat group for technical discussions and updates.
<p align="center">
 <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/wechat.jpg" width="300" alt="WeChat group QR code">
 <br>
 <em>Scan to join our WeChat discussion group</em>
</p>