ASLP-lab commited on
Commit
4aaa430
·
verified ·
1 Parent(s): 04afb46

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +44 -1
README.md CHANGED
@@ -5,4 +5,47 @@ base_model:
5
  - Qwen/Qwen3-Omni-30B-A3B-Instruct
6
  pipeline_tag: audio-classification
7
  ---
8
- A Fine-grained Multi-dimensional Speech understanding model.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
  - Qwen/Qwen3-Omni-30B-A3B-Instruct
6
  pipeline_tag: audio-classification
7
  ---
8
+ Leveraging the multi-dimensional fine-grained annotations produced by our pipeline, we introduce **FM-Speech**, built upon the frontier Qwen3-Omni (30B MoE) architecture.
9
+
10
+ <div align="center">
11
+ <!-- 请在此处放一张精简的模型输入输出示意图 -->
12
+ <!--<img width="500px" src="src/model_io.png" alt="Model Input Output Diagram"/> -->
13
+ <p><i>Input: Raw Speech &emsp; ➔ &emsp; Output: 14-Dimension Fine-Grained Speech Attributes (Structured JSON)</i></p>
14
+ </div>
15
+
16
+ To overcome modality gaps and text-conditioned hallucinations, FM-Speech is trained using a **Progressive Curriculum Fine-Tuning** framework, decoupling complex auditory comprehension into three incremental stages: Warm-up (MCQ/QA) $\rightarrow$ Capability Ramp-up $\rightarrow$ Final Alignment (Full JSON).
17
+
18
+ ### 🚀 Usage & Environment Setup
19
+
20
+ Our model is built upon the Qwen3-Omni architecture. We strongly recommend using **vLLM** for the inference and deployment of FM-Speech.
21
+
22
+ **Step 1: Create a fresh Python environment** to avoid runtime conflicts and incompatibilities.
23
+ ```bash
24
+ conda create -n fmspeech python=3.12
25
+ conda activate fmspeech
26
+ ```
27
+
28
+ **Step 2: Install required packages**
29
+ ```bash
30
+ # Install vLLM (Specifically version 0.13.0)
31
+ pip install vllm==0.13.0
32
+ # Note: If you meet an "Undefined symbol" error while using VLLM_USE_PRECOMPILED=1,
33
+ # please use "pip install -e . -v" to build vLLM from source.
34
+
35
+ # Install Transformers and Accelerate
36
+ pip install transformers==4.57.3
37
+ pip install accelerate
38
+
39
+ # Install Qwen Omni utilities and Flash Attention
40
+ pip install qwen-omni-utils -U
41
+ pip install -U flash-attn --no-build-isolation
42
+ ```
43
+
44
+ **Step 3: Run Inference**
45
+ Prepare a sample audio file and run the inference script to generate the 14-dimension JSON output.
46
+ ```bash
47
+ python infer.py
48
+ ```
49
+ *(See `infer.py` in our repository for detailed loading and inference examples).*
50
+
51
+ ---