| <p align="center"> | |
| <h1>OSUM-Pangu: An Open-Source Multidimension Speech Understanding Foundation Model Built upon OpenPangu on Ascend NPUs</h1> | |
| <p> | |
| Yujie Liao, Xuelong Geng, Hongfei Xue, Shuiyuan Wang, Lei Xie | |
| <p align="center"> | |
| <img src="images/OSUM-Pangu.jpg" width="400"/> | |
| <p> | |
| <p align="center"> | |
| <a href="https://github.com/ASLP-lab/OSUM-Pangu"> Code ｜ <a href="https://arxiv.org/abs/2603.10862"> Paper</a> | |
| </p> | |
| Recent advancements in Speech Large Language Models have significantly enhanced multi-dimensional speech understanding. However, the majority of high-performance frameworks are predominantly optimized for GPU centric ecosystems and proprietary backbones, creating a significant gap for deployment on non-CUDA computing infrastructures. In this paper, we present OSUM-Pangu, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack. By integrating an audio encoder with the openPangu-7B LLM backbone, we successfully implement the entire training and inference pipeline on the Ascend NPU platform. To facilitate efficient task alignment under non-CUDA resource constraints, we adopt a practical training process that sequentially bridges speech perception and user intent recognition. Experimental results demonstrate that OSUM-Pangu achieves task accuracy comparable to mainstream GPU-based models while maintaining robust natural language interaction capabilities. Our work provides a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence. | |
| --- | |
| ## Architecture | |
| The overall architecture of OSUM-Pangu is shown below: | |
| <p align="center"> | |
| <img src="images/architecture1.png" width="80%"/> | |
| <p> | |
| The model mainly consists of three components: | |
| ### 1. Speech Encoder | |
| Whisper-medium | |
| Responsible for extracting speech representations. | |
| ### 2. Adapter | |
| Transforms acoustic features into tokens compatible with the LLM input space. | |
| ### 3. Intent-aware LLM | |
| <p> | |
| <a href="https://huggingface.co/FreedomIntelligence/openPangu-Embedded-7B-V1.1"> openPangu-Embedded-7B-V1.1 </a> | |
| </p> | |
| Responsible for: | |
| - Parsing natural language instructions | |
| - Identifying user intent | |
| - Determining which speech task to execute | |
| --- | |
| ## Training Strategy | |
| We adopt a a three-stage training proces, illustrated below: | |
| <p align="center"> | |
| <img src="images/Strategy.png" width="80%"/> | |
| <p> | |
| ### Stage 1: Speech Understanding Alignment | |
| Goal: Equip the model with multi-task speech understanding capability. | |
| Characteristics: | |
| - Only speech-related modules are trained | |
| - Establish strong acoustic representation ability | |
| --- | |
| ### Stage 2: Intent Understanding | |
| Goal: Enable the model to understand natural language user instructions. | |
| Examples: | |
| Please transcribe this audio. | |
| Analyze the speaker's emotion. | |
| Identify what event happens in the audio. | |
| The model learns: | |
| - Instruction semantic understanding | |
| - Task mapping capability | |
| --- | |
| ### Stage 3: Joint Instruction Tuning | |
| In the final stage, joint training allows the model to: | |
| - Automatically parse user instructions | |
| - Identify task types | |
| - Execute the corresponding speech understanding tasks | |
| Without requiring fixed templates, such as: | |
| What is the emotion of this speech? | |
| Can you transcribe this audio? | |
| What event happens in the audio? | |
| The model can correctly understand and execute all of them. | |
| --- | |
| ## Results | |
| ### Dataset Configuration | |
| Our experiments follow the task definitions of the OSUM framework. To maintain the linguistic reasoning capability of the backbone, we incorporate 2M entries from Alpaca-CoT for text-based interactions, with queries synthesized using CosyVoice 2. To evaluate the model's robustness in real-world scenarios, we utilize an Intent-Instruction Set (IIS) containing over 80k training samples and 4k test prompts, covering diverse colloquial user queries. | |
| --- | |
| ### Multi-task Speech Understanding Performance | |
| OSUM-Pangu demonstrates competitive performance across diverse tasks compared to GPU-based baselines Qwen2-Audio and OSUM, proving the effectiveness of the NPU-based pipeline. | |
| | Task | Model | Public Test Set | Metric | Public Result | | |
| |------------|----------------|---------------------------------------------------------------------------------|--------------|-------------------------------------------------------------------------------| | |
| | **ASR** | Qwen2-Audio | | WER/CER (%) | 8.84 / 8.40 <br> 3.0 / 3.0 / 2.9 <br> **1.6 / 3.6** | | |
| | | OSUM | WenetSpeech(n/m) <br> AISHELL-2(m/i/a) <br> LibriSpeech (c/o) | | 6.46 / **5.34** <br> **2.81 / 2.75 / 2.73** <br> 2.19 / 5.53 | | |
| | | **OSUM-Pangu** | | | 7.40 / 10.49 <br> 3.01 / 2.98 / 2.95 <br> 3.51 / 8.36 | | |
| | **VED** | Qwen2-Audio | VocalSound | ACC (%) | **93.3** | | |
| | | OSUM | | | 82.58 | | |
| | | **OSUM-Pangu** | | | 73.04 | | |
| | **SER** | Qwen2-Audio | MELD-test <br> MER2023 | ACC (%) | 55.3 / -- | | |
| | | OSUM | | | 53.38 / 86.43 | | |
| | | **OSUM-Pangu** | | | 36.40 / **89.19** | | |
| | **SGC** | Qwen2-Audio | Kaggle-CommonVoice test | ACC (%) | 97.25 | | |
| | | OSUM | | | **99.41** | | |
| | | **OSUM-Pangu** | | | 97.48 | | |
| | **SAP** | Qwen2-Audio | Kaggle-CommonVoice test | ACC (%) | 35.53 | | |
| | | OSUM | | | 76.52 | | |
| | | **OSUM-Pangu** | | | **83.31** | | |
| --- | |
| ### Instruction Following Performance (IFR) | |
| Instruction Following Rate (IFR) measures the ability of the model to parse natural language instructions and execute the corresponding tasks. | |
| The metric is defined as: | |
| $$ | |
| IFR = \left( \frac{N_{correct}}{N_{total}} \right) \times 100\% | |
| $$ | |
| where: | |
| - $N_{correct}$ represents the number of correctly executed instructions | |
| - $N_{total}$ represents the total number of evaluation samples | |
| Compared with mainstream open-source models, OSUM-Pangu achieves significantly better performance: | |
| | Model | IFR (\%) | | |
| |---------------------------|-----------| | |
| | Qwen2Audio-Instruct | 71.3 | | |
| | **OSUM-Pangu (Ours)** | **90.2** | | |
| --- | |
| ### Flexibility vs Accuracy | |
| We evaluate whether natural language instructions (NL) degrade performance compared to fixed instructions (FI). | |
| Results show that the model maintains strong flexibility while preserving task accuracy. | |
| | Task | Test | FI | NL | $\Delta$ | | |
| |:---- |:---------------------------- |:-------- |:-------- |:--------- | | |
| | ASR | test-net/librispeech-clean | 7.36/3.64 | 7.40/3.51 | +0.04/-0.13 | | |
| | SER | Test<sub>emotion</sub> | 67.39 | 67.41 | +0.02 | | |
| | SGC | Test<sub>gender</sub> | 97.04 | 96.02 | -1.02 | | |
| | SRWT | Test<sub>align</sub> | 22.39 | 17.52 | -4.87 | | |
| | SSR | Test<sub>style</sub> | 62.79 | 58.05 | -4.74 | | |
| | VED | Test<sub>event</sub> | 77.74 | 73.04 | -4.70 | | |
| | SAP | Test<sub>age</sub> | 71.75 | 72.86 | +0.11 | | |
| --- | |
| Conclusion: | |
| Only minor performance drops appear in relatively niche tasks such as: | |
| - Style recognition | |
| - Event detection | |
| Core tasks such as: | |
| - ASR | |
| - SER | |
| - SAP | |
| remain almost unchanged, validating the effectiveness of the three-stage training process. | |
| --- | |
| ### Speech-to-Text Chat (STTC) Capability | |
| We further evaluate the model in conversational reasoning scenarios. | |
| OSUM-Pangu outperforms GLM-4-Voice on the TriviaQA and WebQ benchmarks. | |
| | Model | LLaMA Q | TriviaQA | Web Q | | |
| |:------------------ |:------: |:-------: |:----: | | |
| | ChatGPT-4o | 71.7 | 69.7 | 51.6 | | |
| | GLM-4-Voice | 50.7 | 26.5 | 15.9 | | |
| | DeepTalk | 59.7 | 27.5 | 23.1 | | |
| | OSUM-EChat | 55.3 | 33.7 | 30.4 | | |
| | **OSUM-Pangu** | 44.6 | 28.9 | 29.5 | | |
| --- | |
| ## How to Use the OSUM-Pangu Framework for Training and Inference | |
| ### Environment Setup | |
| Before starting, please ensure that your device supports **NPU** and the Python environment is properly configured. | |
| We recommend running the code on a Linux system. | |
| If Conda is not installed, please refer to: | |
| https://blog.csdn.net/qq_41636123/article/details/130266232 | |
| ```bash | |
| # Create a new conda environment | |
| conda create -n osum_pangu python=3.10 | |
| conda activate osum_pangu | |
| # Clone the repository | |
| git clone https://github.com/ASLP-lab/OSUM-Pangu.git | |
| cd OSUM-Pangu | |
| # Install dependencies | |
| pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple | |
| ``` | |
| ### Model Download | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| snapshot_download( | |
| repo_id="ASLP-lab/OSUM-Pangu", | |
| local_dir="path", | |
| local_dir_use_symlinks=False, | |
| endpoint="https://hf-mirror.com" | |
| ) | |
| ``` | |
| ### Inference | |
| This project provides batch inference scripts for all tasks under in ：OSUM-Pangu/infer_code: | |
| ```shell | |
| python infer_ASR.py | |
| ``` | |
| ### Training | |
| To ensure a smooth training process, please follow the steps below. | |
| #### 1. Data Preparation | |
| Data can be prepared in three formats: | |
| raw、shard、combine | |
| Recommended: shard format | |
| After preparing the dataset, write the generated data index into the following configuration file: | |
| ```yaml | |
| OSUM-Pangu/conf/data_s2t_tmp.yaml | |
| ``` | |
| #### 2. Start Training | |
| Run the main training script: | |
| ```bash | |
| OSUM-Pangu/train.sh | |
| ``` |