Upload folder using huggingface_hub

.gitattributes

@@ -38,3 +38,5 @@ images/Strategy.png filter=lfs diff=lfs merge=lfs -text
 images/structure.png filter=lfs diff=lfs merge=lfs -text
 images/table1.png filter=lfs diff=lfs merge=lfs -text
 images/table4.png filter=lfs diff=lfs merge=lfs -text
+images/OSUM-Pangu.jpg filter=lfs diff=lfs merge=lfs -text
+images/architecture1.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,48 +1,27 @@
 <p align="center">
-   <h1>I-OSUM-Pangu: Intent-Aware Open-Source Speech Understanding Framework</h1>
+   <h1>OSUM-Pangu: An Open-Source Multidimension Speech Understanding Foundation Model Built upon OpenPangu on Ascend NPUs</h1>
 <p>
 
-Yujie Liao, Xuelong Geng, Shuiyuan Wang, Lei Xie
+Yujie Liao, Xuelong Geng, Hongfei Xue, Shuiyuan Wang, Lei Xie
 
 <p align="center">
-    <img src="images/I-OSUM-Pangu.png" width="400"/>
+    <img src="images/OSUM-Pangu.jpg" width="400"/>
 <p>
 
 <p align="center">
-    <a href="https://github.com/ASLP-lab/I-OSUM-Pangu"> Code</a>
+    <a href="https://github.com/ASLP-lab/OSUM-Pangu"> Code ｜ <a href="https://arxiv.org/abs/2603.10862"> Paper</a>
 </p>
 
-In recent years, the development of large-scale audio-language models has enabled multi-dimensional speech understanding. However, most existing open-source models rely on fixed templates or task tags, while more powerful systems are often closed-source or require massive amounts of training data.
-
-We propose **I-OSUM-Pangu**, an efficient, controllable, and fully open-source speech understanding framework.
-
-The model is built upon:
-
-- Whisper-medium speech encoder (from the Whisper series developed by :contentReference[oaicite:0]{index=0})
-- :contentReference[oaicite:1]{index=1} 7B large language model backbone
-
-The core objective of our framework is to enable the model to:
-
-- Understand user instructions expressed in natural language  
-- Automatically identify user intent  
-- Route the request to the corresponding speech understanding task  
-- Work without relying on fixed prompt templates  
-
-Experimental results show that:
-
-- The Instruction Following Rate (IFR) exceeds **90%**
-- While maintaining comparable task performance with traditional fixed-tag approaches
-
-This project releases both code and model weights, aiming to provide a **reproducible and extensible open-source framework** for speech understanding research.
+Recent advancements in Speech Large Language Models have significantly enhanced multi-dimensional speech understanding. However, the majority of high-performance frameworks are predominantly optimized for GPU centric ecosystems and proprietary backbones, creating a significant gap for deployment on non-CUDA computing infrastructures. In this paper, we present OSUM-Pangu, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack. By integrating an audio encoder with the openPangu-7B LLM backbone, we successfully implement the entire training and inference pipeline on the Ascend NPU platform. To facilitate efficient task alignment under non-CUDA resource constraints, we adopt a practical training process that sequentially bridges speech perception and user intent recognition. Experimental results demonstrate that OSUM-Pangu achieves task accuracy comparable to mainstream GPU-based models while maintaining robust natural language interaction capabilities. Our work provides a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence.
 
 ---
 
 ## Architecture
 
-The overall architecture of I-OSUM-Pangu is shown below:
+The overall architecture of OSUM-Pangu is shown below:
 
 <p align="center">
-    <img src="images/structure.png" width="80%"/>
+    <img src="images/architecture1.png" width="80%"/>
 <p>
 
 The model mainly consists of three components:
@@ -55,7 +34,11 @@ Responsible for extracting speech representations.
 Transforms acoustic features into tokens compatible with the LLM input space.
 
 ### 3. Intent-aware LLM
-OpenPangu-7B
+
+<p>
+    <a href="https://huggingface.co/FreedomIntelligence/openPangu-Embedded-7B-V1.1"> openPangu-Embedded-7B-V1.1 </a>
+</p>
+
 
 Responsible for:
 - Parsing natural language instructions
@@ -66,7 +49,7 @@ Responsible for:
 
 ## Training Strategy
 
-We propose a **Decoupled-then-Integrated Training Strategy**, illustrated below:
+We adopt a a three-stage training proces, illustrated below:
 
 <p align="center">
     <img src="images/Strategy.png" width="80%"/>
@@ -118,15 +101,37 @@ The model can correctly understand and execute all of them.
 
 ---
 
-## Inference Results
+## Results
 
 ### Dataset Configuration
 
-The model is trained on **47,000 hours** of multi-task speech data, covering seven core speech tasks. Additionally, a dedicated dataset is constructed to enhance instruction-following ability.
+Our experiments follow the task definitions of the OSUM framework. To maintain the linguistic reasoning capability of the backbone, we incorporate 2M entries from Alpaca-CoT for text-based interactions, with queries synthesized using CosyVoice 2. To evaluate the model's robustness in real-world scenarios, we utilize an Intent-Instruction Set (IIS) containing over 80k training samples and 4k test prompts, covering diverse colloquial user queries.
 
-<p align="center">
-    <img src="images/table1.png" width="65%"/>
-</p>
+
+---
+
+### Multi-task Speech Understanding Performance
+
+OSUM-Pangu demonstrates competitive performance across diverse tasks compared to GPU-based baselines Qwen2-Audio and OSUM, proving the effectiveness of the NPU-based pipeline.
+
+
+| Task       | Model          | Public Test Set                                                                 | Metric       | Public Result                                                                 |
+|------------|----------------|---------------------------------------------------------------------------------|--------------|-------------------------------------------------------------------------------|
+| **ASR**    | Qwen2-Audio    |                                                                                 | WER/CER (%)  | 8.84 / 8.40 <br> 3.0 / 3.0 / 2.9 <br> **1.6 / 3.6**                           |
+|            | OSUM           | WenetSpeech(n/m) <br> AISHELL-2(m/i/a) <br> LibriSpeech (c/o)                   |              | 6.46 / **5.34** <br> **2.81 / 2.75 / 2.73** <br> 2.19 / 5.53                  |
+|            | **OSUM-Pangu** |                                                                                 |              | 7.40 / 10.49 <br> 3.01 / 2.98 / 2.95 <br> 3.51 / 8.36                         |
+| **VED**    | Qwen2-Audio    | VocalSound                                                                      | ACC (%)      | **93.3**                                                                      |
+|            | OSUM           |                                                                                 |              | 82.58                                                                         |
+|            | **OSUM-Pangu** |                                                                                 |              | 73.04                                                                         |
+| **SER**    | Qwen2-Audio    | MELD-test <br> MER2023                                                          | ACC (%)      | 55.3 / --                                                                     |
+|            | OSUM           |                                                                                 |              | 53.38 / 86.43                                                                 |
+|            | **OSUM-Pangu** |                                                                                 |              | 36.40 / **89.19**                                                             |
+| **SGC**    | Qwen2-Audio    | Kaggle-CommonVoice test                                                         | ACC (%)      | 97.25                                                                         |
+|            | OSUM           |                                                                                 |              | **99.41**                                                                     |
+|            | **OSUM-Pangu** |                                                                                 |              | 97.48                                                                         |
+| **SAP**    | Qwen2-Audio    | Kaggle-CommonVoice test                                                         | ACC (%)      | 35.53                                                                         |
+|            | OSUM           |                                                                                 |              | 76.52                                                                         |
+|            | **OSUM-Pangu** |                                                                                 |              | **83.31**                                                                     |
 
 ---
 
@@ -144,12 +149,13 @@ where:
 
 - $N_{correct}$ represents the number of correctly executed instructions  
 - $N_{total}$ represents the total number of evaluation samples
+Compared with mainstream open-source models, OSUM-Pangu achieves significantly better performance:
 
-Compared with mainstream open-source models, **I-OSUM-Pangu achieves significantly better performance**:
 
-<p align="center">
-    <img src="images/table2.png" width="65%"/>
-</p>
+| Model                     | IFR (\%)  |
+|---------------------------|-----------|
+| Qwen2Audio-Instruct       | 71.3      |
+| **OSUM-Pangu (Ours)**     | **90.2**  |
 
 ---
 
@@ -159,9 +165,18 @@ We evaluate whether natural language instructions (NL) degrade performance compa
 
 Results show that the model maintains strong flexibility while preserving task accuracy.
 
-<p align="center">
-    <img src="images/table3.png" width="65%"/>
-</p>
+
+| Task  | Test                          | FI        | NL        | $\Delta$   |
+|:---- |:---------------------------- |:-------- |:-------- |:--------- |
+| ASR   | test-net/librispeech-clean    | 7.36/3.64 | 7.40/3.51 | +0.04/-0.13 |
+| SER   | Test<sub>emotion</sub>        | 67.39     | 67.41     | +0.02      |
+| SGC   | Test<sub>gender</sub>         | 97.04     | 96.02     | -1.02      |
+| SRWT  | Test<sub>align</sub>          | 22.39     | 17.52     | -4.87      |
+| SSR   | Test<sub>style</sub>          | 62.79     | 58.05     | -4.74      |
+| VED   | Test<sub>event</sub>          | 77.74     | 73.04     | -4.70      |
+| SAP   | Test<sub>age</sub>            | 71.75     | 72.86     | +0.11      |
+
+---
 
 Conclusion:
 
@@ -176,50 +191,29 @@ Core tasks such as:
 - SER
 - SAP
 
-remain almost unchanged, validating the effectiveness of the **Decoupled-then-Integrated strategy**.
+remain almost unchanged, validating the effectiveness of the three-stage training process.
 
 ---
 
-### Multi-task Speech Understanding Performance
-
-On public benchmarks, the model demonstrates competitive performance across multiple tasks, particularly in:
-
-- Age prediction
-- Emotion recognition (MER2023)
-
-<p align="center">
-    <img src="images/table4.png" width="65%"/>
-</p>
-
----
 
 ### Speech-to-Text Chat (STTC) Capability
 
 We further evaluate the model in conversational reasoning scenarios.
 
-I-OSUM-Pangu outperforms GLM-4-Voice on the TriviaQA and WebQ benchmarks.
+OSUM-Pangu outperforms GLM-4-Voice on the TriviaQA and WebQ benchmarks.
 
-<p align="center">
-    <img src="images/table5.png" width="65%"/>
-</p>
-
----
-
-### Ablation Study: Importance of the Decoupled Training Strategy
-
-We compare direct joint training with our decoupled-then-integrated strategy to verify the effectiveness of our core design.
-
-<p align="center">
-    <img src="images/table6.png" width="65%"/>
-</p>
-
-Conclusion:
+| Model               | LLaMA Q | TriviaQA | Web Q |
+|:------------------ |:------: |:-------: |:----: |
+| ChatGPT-4o          | 71.7    | 69.7     | 51.6  |
+| GLM-4-Voice         | 50.7    | 26.5     | 15.9  |
+| DeepTalk            | 59.7    | 27.5     | 23.1  |
+| OSUM-EChat          | 55.3    | 33.7     | 30.4  |
+| **OSUM-Pangu**      | 44.6    | 28.9     | 29.5  |
 
-Text-domain intent pretraining (Stage 2) establishes a strong semantic prior for the model and is crucial for improving instruction-following stability.
 
 ---
 
-## How to Use the I-OSUM-Pangu Framework for Training and Inference
+## How to Use the OSUM-Pangu Framework for Training and Inference
 
 ### Environment Setup
 
@@ -232,12 +226,12 @@ https://blog.csdn.net/qq_41636123/article/details/130266232
 
 ```bash
 # Create a new conda environment
-conda create -n iosum python=3.10
-conda activate iosum
+conda create -n osum_pangu python=3.10
+conda activate osum_pangu
 
 # Clone the repository
-git clone https://github.com/ASLP-lab/I-OSUM-Pangu.git
-cd I-OSUM-Pangu
+git clone https://github.com/ASLP-lab/OSUM-Pangu.git
+cd OSUM-Pangu
 
 # Install dependencies
 pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
@@ -247,16 +241,15 @@ pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
 ```python
 from huggingface_hub import snapshot_download
 
-# 下载Qwen2-Audio-7B模型
 snapshot_download(
-    repo_id="ASLP-lab/I-OSUM-Pangu",
+    repo_id="ASLP-lab/OSUM-Pangu",
     local_dir="path",
     local_dir_use_symlinks=False,
     endpoint="https://hf-mirror.com"
 )
 ```
 ### Inference
-This project provides batch inference scripts for all tasks under in ：I-OSUM-Pangu/infer_code:
+This project provides batch inference scripts for all tasks under in ：OSUM-Pangu/infer_code:
 
 ```shell
 python infer_ASR.py
@@ -272,11 +265,11 @@ Recommended: shard format
 
 After preparing the dataset, write the generated data index into the following configuration file:
 ```yaml
-I-OSUM-Pangu/conf/data_s2t_tmp.yaml
+OSUM-Pangu/conf/data_s2t_tmp.yaml
 ```
 #### 2. Start Training
 
 Run the main training script:
 ```bash
-I-OSUM-Pangu/train.sh
+OSUM-Pangu/train.sh
 ```