--- license: apache-2.0 language: - zh - yue - en pipeline_tag: automatic-speech-recognition library_name: transformers tags: - asr - speech-recognition - chinese - dialect - qwen3-asr - audio base_model: Qwen/Qwen3-ASR-1.7B base_model_relation: finetune ---

**On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin**

Shuiyuan Wang¹ · Bingshen Mu¹ · Pengshen Zhang² · Chengyou Wang¹ · Yujie Liao¹ · Chengdong Liang² · Binbin Zhang² · Qiangze Feng³ · Lei Xie¹

¹ Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University, Xi'an, China
² WeNet Community
³ NEXDATA TECHNOLOGY INC.

[![Paper](https://img.shields.io/badge/Paper-arXiv-blue)](https://arxiv.org/abs/2608.11898) [![GitHub](https://img.shields.io/badge/GitHub-ASLP--lab%2FCN--MultiDialect--ASR-black)](https://github.com/ASLP-lab/CN-MultiDialect-ASR) [![License](https://img.shields.io/badge/License-Apache%202.0-green)](https://huggingface.co/ASLP-lab/CN-MultiDialect-ASR)

This repository hosts the released **CN-MultiDialect-ASR** checkpoint, adapted from [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) with a three-stage pipeline: continual pre-training (CPT), dialect supervised fine-tuning (SFT), and On-Policy Self-Distillation (OPSD). The goal is to improve Chinese dialect recognition **without raising Mandarin CER**. - Paper: [arXiv:2608.11898](https://arxiv.org/abs/2608.11898) - Code, demo, and training scripts: [ASLP-lab/CN-MultiDialect-ASR](https://github.com/ASLP-lab/CN-MultiDialect-ASR)

Overview of the staged adaptation pipeline. Top: base model, CPT, SFT, and OPSD. Bottom: OPSD with student on-policy prefixes, a frozen teacher conditioned on the reference transcript as privileged context, soft targets q_t, and token-level KL.

## Demo Video demo with live waveforms and model transcriptions for Mandarin, English, four core dialects, and 15 ChinaVoices dialects.

## Key Features - **Mandarin–dialect balanced adaptation**: improves Chinese dialect ASR while retaining Mandarin recognition. - **Three-stage pipeline**: CPT strengthens the Chinese ASR foundation, dialect SFT specializes for dialects, and OPSD refines the final checkpoint. - **On-Policy Self-Distillation**: trains on student-decoded prefixes with soft teacher targets, reducing the train–test mismatch of teacher-forced ASR training. - **Drop-in inference**: compatible with the official [`qwen-asr`](https://github.com/QwenLM/Qwen3-ASR) package. ## Quickstart Inference is compatible with [Qwen3-ASR](https://github.com/QwenLM/Qwen3-ASR). We recommend installing the official `qwen-asr` package in a clean environment. ### Environment Setup ```bash conda create -n qwen3-asr python=3.12 -y conda activate qwen3-asr pip install -U qwen-asr ``` For faster inference with the vLLM backend: ```bash pip install -U qwen-asr[vllm] ``` ### Model Download You can load the model directly from Hugging Face, or download it locally first: ```bash # Hugging Face pip install -U "huggingface_hub[cli]" hf download ASLP-lab/CN-MultiDialect-ASR --local-dir ./CN-MultiDialect-ASR # ModelScope (recommended for users in Mainland China) pip install -U modelscope modelscope download --model ASLP-lab/CN-MultiDialect-ASR --local_dir ./CN-MultiDialect-ASR ``` ### Python Inference Load the model with `Qwen3ASRModel.from_pretrained` and call `transcribe`: ```python import torch from qwen_asr import Qwen3ASRModel model = Qwen3ASRModel.from_pretrained( "ASLP-lab/CN-MultiDialect-ASR", # or "./CN-MultiDialect-ASR" for a local path dtype=torch.bfloat16, device_map="cuda:0", # attn_implementation="flash_attention_2", max_inference_batch_size=32, max_new_tokens=256, ) results = model.transcribe( audio="path/to/audio.wav", language="Chinese", # or None for automatic language detection ) print(results[0].language) print(results[0].text) ``` Batch inference is also supported: ```python results = model.transcribe( audio=[ "path/to/mandarin.wav", "path/to/dialect.wav", ], language=["Chinese", "Chinese"], ) for r in results: print(r.language, r.text) ``` For vLLM backend, streaming inference, and forced alignment, see the [Qwen3-ASR repository](https://github.com/QwenLM/Qwen3-ASR). ## Method Overview | Stage | Training data | Goal | Objective | |:-----:|:--------------|:-----|:----------| | `CPT` | Full Mandarin-dialect collection (`~100k` hours) | Build a stronger Chinese ASR foundation | Cross-entropy | | `SFT` | Same sources with higher dialect sampling weight and a small Mandarin anchor | Lower dialect CER | Cross-entropy | | `OPSD` | Dialect refinement subset (`~5k` hours) | Improve dialect recognition without hurting Mandarin | Token-level KL | At inference time, only the student pathway is used. ## Performances ### Dialect Overview

Side-by-side radar of 1-CER on public and internal dialect sets

Higher is better. Left: 5 public dialect sets; right: 18 internal dialects. Both panels use the same radial scale (0.2–1.0). The figure compares the Qwen3-ASR baseline with the released CN-MultiDialect-ASR (OPSD) checkpoint.

### Public Dialect CER (%)

Evaluation set	Dialect	Qwen3-ASR	CN-MultiDialect-ASR
WenetSpeech-Yue Long	Cantonese	9.99	8.80
WenetSpeech-Yue Short	Cantonese	6.93	5.31
WenetSpeech-Chuan Easy	Sichuan	12.38	11.86
WenetSpeech-Chuan Hard	Sichuan	21.79	21.74
WenetSpeech-Wu	Wu	25.74	16.26
Dialect Avg.		15.37	12.79

### Internal Dialect CER (%)

Dialect	Qwen3-ASR	CN-MultiDialect-ASR
Anhui	18.95	13.08
Cantonese	10.06	7.74
Changsha	14.79	10.23
Chaoshan	45.59	25.21
Dongbei	6.45	5.80
Henan	8.46	5.99
Kejia	60.47	28.60
Minnan	30.03	18.59
Nanchang	33.41	15.58
Nanjing	13.37	9.33
Shanxi	28.53	18.69
Shaanxi	9.68	6.28
Shandong	8.78	7.64
Shanghai	15.78	12.07
Sichuan	5.99	5.38
Suzhou	50.35	20.73
Wuhan	11.30	7.59
Xuzhou	6.12	5.04
Internal Avg.	21.01	12.42

### Mandarin CER (%)

Evaluation set	Qwen3-ASR	CN-MultiDialect-ASR
AISHELL-1	1.57	1.38
AISHELL-2	2.79	2.52
KeSpeech	5.11	4.56
SpeechIO-1	0.75	0.86
SpeechIO-2	3.83	3.39
SpeechIO-3	1.39	1.27
Test_Meeting	6.74	6.85
Test_Net	5.46	5.30
Mandarin Avg.	3.46	3.27

## Citation If you use this model, please cite: ```bibtex @misc{wang2026onpolicyselfdistillationmultidialectasr, title={On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin}, author={Shuiyuan Wang and Bingshen Mu and Pengshen Zhang and Chengyou Wang and Yujie Liao and Chengdong Liang and Binbin Zhang and Qiangze Feng and Lei Xie}, year={2026}, eprint={2608.11898}, archivePrefix={arXiv}, primaryClass={eess.AS}, url={https://arxiv.org/abs/2608.11898} } ``` ## License The released model is licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). ## Contact For questions or collaborations, please contact [wangshuiyuan@mail.nwpu.edu.cn](mailto:wangshuiyuan@mail.nwpu.edu.cn). You are also welcome to join our WeChat group for technical discussions and updates.

WeChat group QR code
Scan to join our WeChat discussion group