| # ZipVoice AXERA C++ |
|
|
| ZipVoice TTS 在 AXERA NPU 板卡上的 C++ 推理实现。全链路 axmodel，无 Python 依赖（英文分词除外）。 |
|
|
| ## 性能 (AX650) |
|
|
| ### Distill 模型 |
|
|
| | 场景 | 音频 | 耗时 | RTF | |
| |------|------|------|-----| |
| | 中文句子 | 6.41s | 1.057s | 0.165 | |
| | 中文段落 | 44.97s | 7.357s | 0.164 | |
| | 英文句子 | 6.41s | 1.067s | 0.166 | |
| | 英文段落 | 59.31s | 10.529s | 0.178 | |
|
|
| ### 普通模型 |
|
|
| | 场景 | 音频 | 耗时 | RTF | |
| |------|------|------|-----| |
| | 中文句子 | 6.41s | 5.709s | 0.891 | |
| | 中文段落 | 44.97s | 39.714s | 0.883 | |
| | 英文句子 | 6.41s | 5.693s | 0.888 | |
| | 英文段落 | 59.31s | 56.759s | 0.957 | |
|
|
|
|
|
|
| ## 目录结构 |
|
|
| ``` |
| ├── CMakeLists.txt # 构建配置 |
| ├── zipvoice.cpp # 主程序 |
| ├── build_ax650.sh # 交叉编译脚本 |
| ├── download_bsp.sh # 下载 BSP SDK |
| ├── cmake/ |
| │ └── msp_dependencies.cmake # MSP SDK 配置 |
| ├── toolchains/ |
| │ └── aarch64-none-linux-gnu.toolchain.cmake # 工具链 |
| ├── third_party/kissfft/ # kissfft (vocoder IRFFT) |
| ├── utils/ # 日志/IO/检查 |
| ├── src/ # 核心源码 |
| │ ├── EngineWrapper.* # AX引擎封装 |
| │ ├── tokenizer.* # 中文分词 (41K汉字pinyin映射) |
| │ ├── pinyin_table.hpp # 汉字→拼音表 (自动生成) |
| │ ├── fbank.* # Mel滤波器 (FFT) |
| │ ├── zipvoice_engine.* # 推理引擎 (encoder+decoder4) |
| │ ├── vocoder.* # 声码器 (axmodel+kissfft IRFFT) |
| │ └── wav_writer.hpp # WAV写入 |
| ├── vocoder/ |
| │ └── vocos_full.axmodel # 量化声码器 (15MB) |
| ├── scripts/ # Python辅助脚本 |
| │ ├── gen_cat_tokens.py # 预计算token |
| │ ├── gen_pinyin_table.py # 生成汉字映射表 |
| │ ├── export_vocos_onnx.py # 导出ONNX |
| │ ├── generate_vocoder_calib.py # 校准数据 |
| │ └── quantize_vocoder.sh # pulsar2量化 |
| └── README.md |
| ``` |
|
|
| ## 环境准备 |
|
|
| ```bash |
| # 下载 BSP SDK |
| bash download_bsp.sh |
| |
| # 安装交叉编译器 |
| wget https://developer.arm.com/-/media/Files/downloads/gnu-a/9.2-2019.12/binrel/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu.tar.xz |
| tar -xf gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu.tar.xz |
| # 编译时传 -DTOOLCHAIN_DIR=/path/to/gcc-arm-... 或在 toolchains/ 中修改 |
| ``` |
|
|
| ## 编译 |
|
|
| ```bash |
| mkdir build_ax650 && cd build_ax650 |
| cmake .. \ |
| -DCMAKE_TOOLCHAIN_FILE=../toolchains/aarch64-none-linux-gnu.toolchain.cmake \ |
| -DTOOLCHAIN_DIR=/path/to/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu \ |
| -DCMAKE_INSTALL_PREFIX=../install/ax650 \ |
| -DCMAKE_BUILD_TYPE=Release |
| make -j$(nproc) && make install |
| ``` |
|
|
| ## 板端推理 |
|
|
| 所有命令在 repo 根目录（`ZipVoice.AXERA/`）下执行。 |
|
|
| ### 中文 |
|
|
| ```bash |
| # 普通模型 — 句子 |
| cpp/install/ax650/zipvoice_axera \ |
| --model-dir ./models/zipvoice_ax650 \ |
| --token-file ./resources/zipvoice_hf/zipvoice/tokens.txt \ |
| --prompt-wav ./assets/moss_prompts/zh_1_4p5s.wav \ |
| --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \ |
| --text "今天午后天气很好，我打开窗户，听见远处有人聊天，水杯也轻轻晃了一下。" \ |
| --vocoder-model ./cpp/vocoder/vocos_full.axmodel \ |
| --output-wav output.wav --seed 42 |
| ``` |
|
|
| ```text |
| 推理结果: |
| Segments: 1 |
| Audio duration: 6.41 s |
| Total time: 5.71 s |
| RTF (端到端): 0.891 |
| ``` |
|
|
| ```bash |
| # 蒸馏模型 — 句子 (约3倍速) |
| cpp/install/ax650/zipvoice_axera \ |
| --model-dir ./models/zipvoice_distill_ax650 \ |
| --token-file ./resources/zipvoice_hf/zipvoice/tokens.txt \ |
| --prompt-wav ./assets/moss_prompts/zh_1_4p5s.wav \ |
| --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \ |
| --text "今天午后天气很好，我打开窗户，听见远处有人聊天，水杯也轻轻晃了一下。" \ |
| --vocoder-model ./cpp/vocoder/vocos_full.axmodel \ |
| --output-wav output.wav --seed 42 |
| ``` |
|
|
| ```text |
| 推理结果: |
| Segments: 1 |
| Audio duration: 6.41 s |
| Total time: 1.06 s |
| RTF (端到端): 0.165 |
| ``` |
|
|
| ```bash |
| # 普通模型 — 段落 |
| cpp/install/ax650/zipvoice_axera \ |
| --model-dir ./models/zipvoice_ax650 \ |
| --token-file ./resources/zipvoice_hf/zipvoice/tokens.txt \ |
| --prompt-wav ./assets/moss_prompts/zh_1_4p5s.wav \ |
| --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \ |
| --text-file ./assets/paragraphs/zh_ginkgo.txt \ |
| --vocoder-model ./cpp/vocoder/vocos_full.axmodel \ |
| --output-wav output.wav --seed 42 |
| ``` |
|
|
| ```text |
| 推理结果: |
| Segments: 7 |
| Audio duration: 44.97 s |
| Total time: 39.71 s |
| RTF (端到端): 0.883 |
| ``` |
|
|
| ```bash |
| # 蒸馏模型 — 段落 |
| cpp/install/ax650/zipvoice_axera \ |
| --model-dir ./models/zipvoice_distill_ax650 \ |
| --token-file ./resources/zipvoice_hf/zipvoice/tokens.txt \ |
| --prompt-wav ./assets/moss_prompts/zh_1_4p5s.wav \ |
| --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \ |
| --text-file ./assets/paragraphs/zh_ginkgo.txt \ |
| --vocoder-model ./cpp/vocoder/vocos_full.axmodel \ |
| --output-wav output.wav --seed 42 |
| ``` |
|
|
| ```text |
| 推理结果: |
| Segments: 7 |
| Audio duration: 44.97 s |
| Total time: 7.38 s |
| RTF (端到端): 0.164 |
| ``` |
|
|
| ### 英文 (加 `--repo-dir .` 自动调 Python 分词器) |
|
|
| ```bash |
| # 普通模型 — 句子 |
| cpp/install/ax650/zipvoice_axera \ |
| --model-dir ./models/zipvoice_ax650 \ |
| --prompt-wav ./assets/moss_prompts/en_4_4p5s.wav \ |
| --prompt-text "This is almost twice the current industry production level per train." \ |
| --text "This morning, a small train left the station, carrying sleepy passengers toward a bright coastal town." \ |
| --repo-dir . \ |
| --vocoder-model ./cpp/vocoder/vocos_full.axmodel \ |
| --output-wav output_en.wav --seed 42 |
| ``` |
|
|
| ```text |
| 推理结果: |
| Segments: 1 |
| Audio duration: 6.41 s |
| Total time: 5.69 s |
| RTF (端到端): 0.888 |
| ``` |
|
|
| ```bash |
| # 蒸馏模型 — 句子 |
| cpp/install/ax650/zipvoice_axera \ |
| --model-dir ./models/zipvoice_distill_ax650 \ |
| --prompt-wav ./assets/moss_prompts/en_4_4p5s.wav \ |
| --prompt-text "This is almost twice the current industry production level per train." \ |
| --text "This morning, a small train left the station, carrying sleepy passengers toward a bright coastal town." \ |
| --repo-dir . \ |
| --vocoder-model ./cpp/vocoder/vocos_full.axmodel \ |
| --output-wav output_en.wav --seed 42 |
| ``` |
|
|
| ```text |
| 推理结果: |
| Segments: 1 |
| Audio duration: 6.41 s |
| Total time: 1.06 s |
| RTF (端到端): 0.166 |
| ``` |
|
|
| ```bash |
| # 普通模型 — 段落 |
| cpp/install/ax650/zipvoice_axera \ |
| --model-dir ./models/zipvoice_ax650 \ |
| --prompt-wav ./assets/moss_prompts/en_4_4p5s.wav \ |
| --prompt-text "This is almost twice the current industry production level per train." \ |
| --text-file ./assets/paragraphs/en_scavenger.txt \ |
| --repo-dir . \ |
| --vocoder-model ./cpp/vocoder/vocos_full.axmodel \ |
| --output-wav output_en.wav --seed 42 |
| ``` |
|
|
| ```text |
| 推理结果: |
| Segments: 10 |
| Audio duration: 59.31 s |
| Total time: 56.76 s |
| RTF (端到端): 0.957 |
| ``` |
|
|
| ```bash |
| # 蒸馏模型 — 段落 |
| cpp/install/ax650/zipvoice_axera \ |
| --model-dir ./models/zipvoice_distill_ax650 \ |
| --prompt-wav ./assets/moss_prompts/en_4_4p5s.wav \ |
| --prompt-text "This is almost twice the current industry production level per train." \ |
| --text-file ./assets/paragraphs/en_scavenger.txt \ |
| --repo-dir . \ |
| --vocoder-model ./cpp/vocoder/vocos_full.axmodel \ |
| --output-wav output_en.wav --seed 42 |
| ``` |
|
|
| ```text |
| 推理结果: |
| Segments: 10 |
| Audio duration: 59.31 s |
| Total time: 10.56 s |
| RTF (端到端): 0.178 |
| ``` |
|
|
| ## 命令行参数 |
|
|
| | 参数 | 说明 | 默认值 | |
| |------|------|--------| |
| | `--model-dir` | 模型目录 | 必填 | |
| | `--token-file` | tokens.txt (中文) | 可选 | |
| | `--prompt-wav` | 提示音频 | 必填 | |
| | `--prompt-text` | 提示文本 | 必填 | |
| | `--text` / `--text-file` | 合成文本 | 二选一 | |
| | `--repo-dir` | repo根目录 (英文分词) | 可选 | |
| | `--vocoder-model` | vocos axmodel 路径 | 必填 | |
| | `--output-wav` | 输出WAV | output.wav | |
| | `--num-step` | 采样步数 | 10 (distill: 4) | |
| | `--speed` | 语速 | 1.0 | |
| | `--seed` | 随机种子 | 42 | |
|
|
| ## 重新量化 vocoder |
|
|
| ```bash |
| cd cpp/scripts |
| python3 export_vocos_onnx.py # 导出ONNX |
| python3 generate_vocoder_calib.py # 生成校准数据 |
| source /path/to/npu_dev && bash quantize_vocoder.sh # 量化 |
| ``` |
|
|
| ## 实现要点 |
|
|
| - **中文分词**: C++ 内置, 基于 pypinyin 的 41K 汉字→拼音映射表, 0ms 开销 |
| - **英文分词**: Python daemon 持久进程, 首次 import ~8s, 后续每次 ~10ms |
| - **长文本分段**: 对齐 Python `text_processing.build_segments`, 自动按 token/帧数约束切段 |
| - **声码器**: vocos axmodel (NPU) + kissfft IRFFT (CPU), 比手写逐点版快 ~30x |
| - **RTF 口径**: 仅计入增量推理 (tokenizer + encoder + decoder + vocoder), 不含一次性初始化 |
|
|
| ## 许可 |
|
|
| MIT License |
|
|
