README.md

# ZipVoice.AXERA

[ZipVoice]((https://github.com/k2-fsa/ZipVoice)) AXERA 板端推理 demo。

## 功能

- 支持中文和英文语音生成。
- 支持语音克隆。
- 支持 ZipVoice、ZipVoice Distill

## 模型说明

ZipVoice Distill 是 ZipVoice 的蒸馏版本，主要优势是在较小性能损失下提升推理速度。初步测试，AX650 ZipVoice Distill 在长文本场景下相比基础版模型约有 3 倍速度提升，RTF 在 0.3 左右，效果没有明显下降。

AX630C 版本当前推理结果差，RTF 约为 1.5 左右，需要继续调优。

## 模型转换

模型量化参考:

- [Pulsar2 Docs — How to Convert ONNX to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html)
- [export / quantization Project](https://github.com/AXERA-TECH/ZipVoice.AXERA)

## 支持平台

- AX650
  - AX650 demo 板
  - [M4N-Dock（爱芯派Pro）](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
  - [M.2 Accelerator Card](https://docs.m5stack.com/zh_CN/ai_hardware/LLM-8850_Card)


## 目录结构

```text
ZipVoice.AXERA/
├── assets/
│   ├── moss_prompts/
│   └── paragraphs/
├── models/
│   ├── zipvoice_ax650/
│   ├── zipvoice_distill_ax650/
│   └── zipvoice_distill_ax630C/
├── resources/
│   ├── vocos-mel-24khz/
│   └── zipvoice_hf/
├── scripts/
├── infer_zipvoice_axera.py
├── requirements.txt
└── README.md
```

## 环境

安装 pyaxengine：

```bash
pip3 install axengine-x.x.x-py3-none-any.whl
```

安装依赖：

```bash
conda create -n ZipVoice python=3.10
conda activate ZipVoice
pip3 install -r requirements.txt
```

## 推理命令

进入目录：

```bash
cd ZipVoice.AXERA
```

### AX650 ZipVoice

中文句子：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_ax650 \
  --text "今天午后天气很好，我打开窗户，听见远处有人聊天，水杯也轻轻晃了一下。" \
  --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \
  --prompt-wav assets/moss_prompts/zh_1_4p5s.wav \
  --output-wav outputs/zh_sentence_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 5.781s
生成语音时长: 6.411s
RTF: 0.9018
```

音频：[outputs/zh_sentence_ax650.wav](outputs/zh_sentence_ax650.wav)  
提示音：[assets/moss_prompts/zh_1_4p5s.wav](assets/moss_prompts/zh_1_4p5s.wav)

英文句子：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_ax650 \
  --text "This morning, a small train left the station, carrying sleepy passengers toward a bright coastal town." \
  --prompt-text "This is almost twice the current industry production level per train." \
  --prompt-wav assets/moss_prompts/en_4_4p5s.wav \
  --output-wav outputs/en_sentence_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 5.711s
生成语音时长: 6.411s
RTF: 0.8909
```

音频：[outputs/en_sentence_ax650.wav](outputs/en_sentence_ax650.wav)  
提示音：[assets/moss_prompts/en_4_4p5s.wav](assets/moss_prompts/en_4_4p5s.wav)

中文段落：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_ax650 \
  --text-file assets/paragraphs/zh_ginkgo.txt \
  --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \
  --prompt-wav assets/moss_prompts/zh_1_4p5s.wav \
  --output-wav outputs/zh_long_paragraph_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 40.292s
生成语音时长: 44.744s
RTF: 0.9005
```

音频：[outputs/zh_long_paragraph_ax650.wav](outputs/zh_long_paragraph_ax650.wav)  
提示音：[assets/moss_prompts/zh_1_4p5s.wav](assets/moss_prompts/zh_1_4p5s.wav)

英文段落：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_ax650 \
  --text-file assets/paragraphs/en_scavenger.txt \
  --prompt-text "This is almost twice the current industry production level per train." \
  --prompt-wav assets/moss_prompts/en_4_4p5s.wav \
  --output-wav outputs/en_long_paragraph_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 62.161s
生成语音时长: 64.749s
RTF: 0.9600
```

音频：[outputs/en_long_paragraph_ax650.wav](outputs/en_long_paragraph_ax650.wav)  
提示音：[assets/moss_prompts/en_4_4p5s.wav](assets/moss_prompts/en_4_4p5s.wav)

### AX650 ZipVoice Distill

中文句子：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_distill_ax650 \
  --text "今天午后天气很好，我打开窗户，听见远处有人聊天，水杯也轻轻晃了一下。" \
  --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \
  --prompt-wav assets/moss_prompts/zh_1_4p5s.wav \
  --output-wav outputs/zh_sentence_distill_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 1.992s
生成语音时长: 6.411s
RTF: 0.3107
```

音频：[outputs/zh_sentence_distill_ax650.wav](outputs/zh_sentence_distill_ax650.wav)  
提示音：[assets/moss_prompts/zh_1_4p5s.wav](assets/moss_prompts/zh_1_4p5s.wav)

英文句子：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_distill_ax650 \
  --text "This morning, a small train left the station, carrying sleepy passengers toward a bright coastal town." \
  --prompt-text "This is almost twice the current industry production level per train." \
  --prompt-wav assets/moss_prompts/en_4_4p5s.wav \
  --output-wav outputs/en_sentence_distill_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 2.045s
生成语音时长: 6.411s
RTF: 0.3189
```

音频：[outputs/en_sentence_distill_ax650.wav](outputs/en_sentence_distill_ax650.wav)  
提示音：[assets/moss_prompts/en_4_4p5s.wav](assets/moss_prompts/en_4_4p5s.wav)

中文段落：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_distill_ax650 \
  --text-file assets/paragraphs/zh_ginkgo.txt \
  --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \
  --prompt-wav assets/moss_prompts/zh_1_4p5s.wav \
  --output-wav outputs/zh_long_paragraph_distill_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 13.457s
生成语音时长: 44.744s
RTF: 0.3008
```

音频：[outputs/zh_long_paragraph_distill_ax650.wav](outputs/zh_long_paragraph_distill_ax650.wav)  
提示音：[assets/moss_prompts/zh_1_4p5s.wav](assets/moss_prompts/zh_1_4p5s.wav)

英文段落：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_distill_ax650 \
  --text-file assets/paragraphs/en_scavenger.txt \
  --prompt-text "This is almost twice the current industry production level per train." \
  --prompt-wav assets/moss_prompts/en_4_4p5s.wav \
  --output-wav outputs/en_long_paragraph_distill_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 19.715s
生成语音时长: 64.749s
RTF: 0.3045
```

音频：[outputs/en_long_paragraph_distill_ax650.wav](outputs/en_long_paragraph_distill_ax650.wav)  
提示音：[assets/moss_prompts/en_4_4p5s.wav](assets/moss_prompts/en_4_4p5s.wav)


## 参数说明

- `--model-name`：选择模型目录。可选 `zipvoice_ax650`、`zipvoice_distill_ax650`、`zipvoice_distill_ax630C`。
- `--prompt-wav`：参考音频，用于控制音色，建议 3-5s。
- `--prompt-text`：参考音频对应文本，必须尽量和 `prompt-wav` 内容一致。
- `--num-step`：采样步数。默认从模型目录的 `runtime_config.json` 读取。
- `--max-feat-len`：decoder 固定 feature 长度，当前模型均为 1024。

## 参考

- [ZipVoice](https://github.com/k2-fsa/ZipVoice)