Update README.md
Browse files
README.md
CHANGED
|
@@ -5,4 +5,47 @@ base_model:
|
|
| 5 |
- Qwen/Qwen3-Omni-30B-A3B-Instruct
|
| 6 |
pipeline_tag: audio-classification
|
| 7 |
---
|
| 8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
- Qwen/Qwen3-Omni-30B-A3B-Instruct
|
| 6 |
pipeline_tag: audio-classification
|
| 7 |
---
|
| 8 |
+
Leveraging the multi-dimensional fine-grained annotations produced by our pipeline, we introduce **FM-Speech**, built upon the frontier Qwen3-Omni (30B MoE) architecture.
|
| 9 |
+
|
| 10 |
+
<div align="center">
|
| 11 |
+
<!-- 请在此处放一张精简的模型输入输出示意图 -->
|
| 12 |
+
<!--<img width="500px" src="src/model_io.png" alt="Model Input Output Diagram"/> -->
|
| 13 |
+
<p><i>Input: Raw Speech   ➔   Output: 14-Dimension Fine-Grained Speech Attributes (Structured JSON)</i></p>
|
| 14 |
+
</div>
|
| 15 |
+
|
| 16 |
+
To overcome modality gaps and text-conditioned hallucinations, FM-Speech is trained using a **Progressive Curriculum Fine-Tuning** framework, decoupling complex auditory comprehension into three incremental stages: Warm-up (MCQ/QA) $\rightarrow$ Capability Ramp-up $\rightarrow$ Final Alignment (Full JSON).
|
| 17 |
+
|
| 18 |
+
### 🚀 Usage & Environment Setup
|
| 19 |
+
|
| 20 |
+
Our model is built upon the Qwen3-Omni architecture. We strongly recommend using **vLLM** for the inference and deployment of FM-Speech.
|
| 21 |
+
|
| 22 |
+
**Step 1: Create a fresh Python environment** to avoid runtime conflicts and incompatibilities.
|
| 23 |
+
```bash
|
| 24 |
+
conda create -n fmspeech python=3.12
|
| 25 |
+
conda activate fmspeech
|
| 26 |
+
```
|
| 27 |
+
|
| 28 |
+
**Step 2: Install required packages**
|
| 29 |
+
```bash
|
| 30 |
+
# Install vLLM (Specifically version 0.13.0)
|
| 31 |
+
pip install vllm==0.13.0
|
| 32 |
+
# Note: If you meet an "Undefined symbol" error while using VLLM_USE_PRECOMPILED=1,
|
| 33 |
+
# please use "pip install -e . -v" to build vLLM from source.
|
| 34 |
+
|
| 35 |
+
# Install Transformers and Accelerate
|
| 36 |
+
pip install transformers==4.57.3
|
| 37 |
+
pip install accelerate
|
| 38 |
+
|
| 39 |
+
# Install Qwen Omni utilities and Flash Attention
|
| 40 |
+
pip install qwen-omni-utils -U
|
| 41 |
+
pip install -U flash-attn --no-build-isolation
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
**Step 3: Run Inference**
|
| 45 |
+
Prepare a sample audio file and run the inference script to generate the 14-dimension JSON output.
|
| 46 |
+
```bash
|
| 47 |
+
python infer.py
|
| 48 |
+
```
|
| 49 |
+
*(See `infer.py` in our repository for detailed loading and inference examples).*
|
| 50 |
+
|
| 51 |
+
---
|