Instructions to use moonbox-dev/VoxCPM2-ONNX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- VoxCPM
How to use moonbox-dev/VoxCPM2-ONNX with VoxCPM:
import soundfile as sf from voxcpm import VoxCPM model = VoxCPM.from_pretrained("moonbox-dev/VoxCPM2-ONNX") wav = model.generate( text="VoxCPM is an innovative end-to-end TTS model from ModelBest, designed to generate highly expressive speech.", prompt_wav_path=None, # optional: path to a prompt speech for voice cloning prompt_text=None, # optional: reference text cfg_value=2.0, # LM guidance on LocDiT, higher for better adherence to the prompt, but maybe worse inference_timesteps=10, # LocDiT inference timesteps, higher for better result, lower for fast speed normalize=True, # enable external TN tool denoise=True, # enable external Denoise tool retry_badcase=True, # enable retrying mode for some bad cases (unstoppable) retry_badcase_max_times=3, # maximum retrying times retry_badcase_ratio_threshold=6.0, # maximum length restriction for bad case detection (simple but effective), it could be adjusted for slow pace speech ) sf.write("output.wav", wav, 16000) print("saved: output.wav") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from moonbox-dev/VoxCPM2-ONNX: direct link, hf CLI and curl.
- Browser
- Download file 24.7 kB
-
https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/resolve/main/README.md
- Command line
-
hf download hf://moonbox-dev/VoxCPM2-ONNX/README.md
-
curl -L -o README.md https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/resolve/main/README.md
24.7 kB
| base_model: openbmb/VoxCPM2 | |
| base_model_relation: quantized | |
| license: apache-2.0 | |
| pipeline_tag: text-to-speech | |
| library_name: onnx | |
| tags: | |
| - onnx | |
| - moonbox | |
| - onnxruntime | |
| - voxcpm | |
| - voxcpm2 | |
| - text-to-speech | |
| - voice-cloning | |
| - 48khz | |
| # VoxCPM2 ONNX | |
| ONNX builds of [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2), made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each | |
| precision is one folder holding the graphs and a small Python `pipeline.py` that does the rest (features, prompt, | |
| sampling loop); what the graphs take and return is under *How to run*. | |
| ## Files | |
| The last column is the benchmark on Moonbox's TTS set (voice cloning of English and Chinese sentences from a | |
| LibriSpeech and an AISHELL-1 prompt; details under *How to run*): error rate of a Whisper transcript against the text, | |
| speaker similarity to the prompt and DNSMOS, against the original PyTorch model. | |
| | Precision | File | Size | sha256 | Benchmark | | |
| |---|---|---|---|---| | |
| | fp16 | [`fp16/base_lm.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/base_lm.onnx) | 17.3 MB | `3a5d89c65ba3` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/base_lm.onnx.data`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/base_lm.onnx.data) | 2.88 GB | `42ac51061af3` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/estimator.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/estimator.onnx) | 424.6 MB | `057b45e894b4` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/feat_encoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/feat_encoder.onnx) | 436.9 MB | `38182a733bf8` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/fsq.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/fsq.onnx) | 4.2 MB | `78076e951846` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/fusion.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/fusion.onnx) | 16.8 MB | `39916677f91b` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/heads.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/heads.onnx) | 16.8 MB | `19d9f4f6d250` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/multichar_tokens.json`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/multichar_tokens.json) | 0.2 MB | `43263698b2c5` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/pipeline.py`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/pipeline.py) | 9.5 kB | `6bee96bae464` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/residual_lm.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/residual_lm.onnx) | 0.1 MB | `55248bc9d387` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/residual_lm.onnx.data`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/residual_lm.onnx.data) | 822.2 MB | `23082e0ccbff` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/text_embed.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/text_embed.onnx) | 300.8 MB | `48be9c770f79` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/tokenizer.json`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/tokenizer.json) | 3.7 MB | `f8984687e4a9` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/vae_decoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/vae_decoder.onnx) | 91.8 MB | `e3205805aa8d` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | fp16 | [`fp16/vae_encoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/fp16/vae_encoder.onnx) | 96.3 MB | `f912268155d7` | WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA) | | |
| | int8 | [`int8/base_lm.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/base_lm.onnx) | 1.43 GB | `e498cb38af6d` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/estimator.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/estimator.onnx) | 213.5 MB | `890a11ed3561` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/feat_encoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/feat_encoder.onnx) | 244.7 MB | `0d3b35cac7f3` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/fsq.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/fsq.onnx) | 2.1 MB | `0e53a24ebbed` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/fusion.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/fusion.onnx) | 8.4 MB | `2ce552d569a2` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/heads.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/heads.onnx) | 8.4 MB | `9a60167d9bac` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/multichar_tokens.json`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/multichar_tokens.json) | 0.2 MB | `43263698b2c5` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/pipeline.py`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/pipeline.py) | 9.5 kB | `6bee96bae464` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/residual_lm.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/residual_lm.onnx) | 403.4 MB | `e557865390b8` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/text_embed.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/text_embed.onnx) | 150.7 MB | `06477af2a9f1` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/tokenizer.json`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/tokenizer.json) | 3.7 MB | `f8984687e4a9` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/vae_decoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/vae_decoder.onnx) | 46.3 MB | `126eba421426` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int8 | [`int8/vae_encoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int8/vae_encoder.onnx) | 48.4 MB | `b5b89a6b44be` | WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA) | | |
| | int4 | [`int4/base_lm.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/base_lm.onnx) | 768.1 MB | `40c52af0269d` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/estimator.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/estimator.onnx) | 213.5 MB | `890a11ed3561` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/feat_encoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/feat_encoder.onnx) | 244.7 MB | `0d3b35cac7f3` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/fsq.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/fsq.onnx) | 2.1 MB | `0e53a24ebbed` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/fusion.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/fusion.onnx) | 8.4 MB | `2ce552d569a2` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/heads.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/heads.onnx) | 8.4 MB | `9a60167d9bac` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/multichar_tokens.json`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/multichar_tokens.json) | 0.2 MB | `43263698b2c5` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/pipeline.py`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/pipeline.py) | 9.5 kB | `6bee96bae464` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/residual_lm.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/residual_lm.onnx) | 214.6 MB | `771cdf95aaf8` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/text_embed.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/text_embed.onnx) | 150.7 MB | `06477af2a9f1` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/tokenizer.json`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/tokenizer.json) | 3.7 MB | `f8984687e4a9` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/vae_decoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/vae_decoder.onnx) | 46.3 MB | `126eba421426` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| | int4 | [`int4/vae_encoder.onnx`](https://huggingface.co/moonbox-dev/VoxCPM2-ONNX/blob/main/int4/vae_encoder.onnx) | 48.4 MB | `b5b89a6b44be` | WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU | | |
| ## How to run | |
| VoxCPM2 (OpenBMB's tokenizer-free speech generator: voice cloning from a reference recording and its transcript, | |
| English and Chinese, 48 kHz output) as ten ONNX graphs plus `pipeline.py`, a numpy runtime that reproduces voxcpm | |
| 2.0.3's `VoxCPM.generate(text, prompt_wav_path, prompt_text, reference_wav_path)` around them: | |
| - `base_lm.onnx` / `residual_lm.onnx`: the two MiniCPM4 language models (28 and 8 layers), with the KV cache as | |
| inputs and outputs: `inputs_embeds` [1, S, 2048], `past_key` / `past_value` [layers, 1, 2, P, 128] -> the final | |
| normed `hidden` [1, S, 2048] and `present_key` / `present_value` [layers, 1, 2, P + S, 128]. | |
| - `estimator.onnx`: the local diffusion transformer that turns each LM step into one patch of 4 latent frames. | |
| `x`, `cond` [2, 64, 4], `mu` [2, 2048], `t`, `dt` [2] -> velocity [2, 64, 4] (conditional and unconditional rows, for | |
| classifier-free guidance). | |
| - `feat_encoder.onnx` (latent patches -> LM inputs), `text_embed.onnx`, `fsq.onnx` (the scalar quantizer on the LM | |
| output), `fusion.onnx` (LM output + audio embedding -> residual LM input), `heads.onnx` (-> the DiT condition and | |
| the stop logits). | |
| - `vae_encoder.onnx` / `vae_decoder.onnx`: the audio VAE. 16 kHz audio [1, 1, N] -> latents [1, 64, N / 640], and | |
| latents [1, 64, T] -> 48 kHz audio [1, 1, 1920 T]. | |
| `pipeline.py` builds the original's prompt (the reference audio as a reference prefix and as the prompt the speech | |
| continues from, then the prompt transcript and the text), runs the LM / DiT loop with 10 Euler steps, classifier-free | |
| guidance 2.0 (CFG-zero*) and the original's length limit (6 x the text's tokens + 10 patches), and decodes. It needs | |
| numpy, onnxruntime (or onnxruntime-gpu), tokenizers, soundfile and soxr: | |
| ```bash | |
| python fp16/pipeline.py "The quick brown fox jumps over the lazy dog." out.wav --ref voice.wav \ | |
| --ref-text "Exactly what is said in voice.wav." --cuda --seed 0 | |
| ``` | |
| ```python | |
| import sys, soundfile as sf | |
| sys.path.insert(0, "fp16") | |
| import pipeline | |
| tts = pipeline.VoxCPM2("fp16", "cuda") | |
| ref, sr = sf.read("voice.wav", dtype="float32") | |
| wav, out_sr = tts("Text to say.", ref, sr, "Transcript of voice.wav.", seed=0) # 48 kHz | |
| ``` | |
| The diffusion noise comes from numpy's generator, so a seed gives the same speech every run here, not the same speech | |
| as PyTorch with that seed. The original's text normalisation and its retry on overlong output are off by default and | |
| not included; pass normalised text. | |
| **Precisions.** `fp16/`: both language models in fp16, every other graph fp16 weights with fp32 math (the diffusion | |
| transformer, the local encoder and the VAE run in fp32 in the original). `int8/`: the language models 8-bit MatMulNBits | |
| (block 32), every other graph weight-only int8, the text embedding int8 per row. `int4/`: the language models 4-bit | |
| MatMulNBits (block 32), the rest as `int8/`. No bf16 build. These are GPU builds: `int4/` on a laptop CPU runs about | |
| 33 times slower than real time (measured on 7 clips). | |
| **Benchmark.** The lab's TTS set: 6 English sentences cloned from a LibriSpeech test-clean prompt and 4 Chinese | |
| sentences from an AISHELL-1 prompt, each with seeds 0-3 (40 clips per build), the original's settings (prompt = | |
| reference = the set's prompt, CFG 2.0, 10 steps, no retry). Sampled speech can't be compared sample by sample, so each | |
| build is judged as the original is: Whisper large-v3-turbo's WER (English) or CER (Chinese) against the text, | |
| WavLM-base-plus-sv speaker similarity to the prompt, and DNSMOS (P.835 overall). The original is voxcpm 2.0.3 in bf16 | |
| (the checkpoint's dtype) on CUDA. Pass (the TTS threshold used across these repositories): per language, error rate | |
| at most 2 points above the original's, similarity at most 0.03 below, DNSMOS at most 0.2 below. Real-time factor | |
| (processing time / audio time) on an RTX 3080 Laptop GPU, whole pipeline. | |
| | | English WER | similarity | DNSMOS | Chinese CER | similarity | DNSMOS | Real-time factor | | |
| |---|---|---|---|---|---|---|---| | |
| | original (PyTorch bf16, CUDA) | 0.0 % | 0.976 | 3.07 | 0.0 % | 0.965 | 2.29 | 2.2 | | |
| | `fp16/` (CUDA) | 0.3 % | 0.973 | 3.06 | 0.0 % | 0.966 | 2.10 | 0.85 | | |
| | `int8/` (CUDA) | 0.0 % | 0.973 | 2.99 | 0.3 % | 0.968 | 2.15 | 1.7 | | |
| | `int4/` (CUDA) | 0.3 % | 0.976 | 3.01 | 0.0 % | 0.969 | 2.11 | 1.7 | | |
| DNSMOS of this model's output varies a lot from sample to sample: the original itself scored 3.17 / 2.36 on seeds 0-1 | |
| and 3.07 / 2.29 on seeds 0-3 here (and 2.93 / 2.28 in fp32 on another machine), so the margins above are within that | |
| spread rather than measured differences between builds. | |
| ## How these were made | |
| From [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) at revision `7936080b41`. The graphs were exported from the original PyTorch modules with | |
| torch.onnx (opset 17 / 18) and checked against them; language-model decoders were built with onnxruntime-genai's | |
| model builder (inputs_embeds in, KV cache in and out). Weight-only quantization: int8 is a | |
| per-output-channel int8 weight behind DequantizeLinear (or 8-bit MatMulNBits in a text decoder), int4 is ONNX | |
| Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was | |
| benchmarked against the original before upload (pass, per language: error rate at most 2 points above the | |
| original's, speaker similarity at most 0.03 below, DNSMOS at most 0.2 below), and the `sha256` column above is what | |
| the files must hash to. | |
| ## License | |
| Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this | |
| repository, and it governs your use of these files. The original model card follows unchanged. | |
| --- | |
| # Original model card | |
| # VoxCPM2 | |
| **VoxCPM2** is a tokenizer-free, diffusion autoregressive Text-to-Speech model β **2B parameters**, **30 languages**, **48kHz** audio output, trained on over **2 million hours** of multilingual speech data. | |
| [](https://github.com/OpenBMB/VoxCPM) | |
| [](https://voxcpm.readthedocs.io/en/latest/) | |
| [](https://huggingface.co/spaces/OpenBMB/VoxCPM-Demo) | |
| [](https://openbmb.github.io/voxcpm2-demopage) | |
| [](https://discord.gg/KZUx7tVNwz) | |
| [](https://applink.feishu.cn/client/chat/chatter/add_by_link?link_token=acds0b9d-23d8-4d7e-b696-d200f3e22a7f) [](https://openbmb.cn/docs/) | |
| ## Highlights | |
| - π **30-Language Multilingual** β No language tag needed; input text in any supported language directly | |
| - π¨ **Voice Design** β Generate a novel voice from a natural-language description alone (gender, age, tone, emotion, paceβ¦); no reference audio required | |
| - ποΈ **Controllable Cloning** β Clone any voice from a short clip, with optional style guidance to steer emotion, pace, and expression while preserving timbre | |
| - ποΈ **Ultimate Cloning** β Provide reference audio + its transcript for audio-continuation cloning; every vocal nuance faithfully reproduced | |
| - π **48kHz Studio-Quality Output** β Accepts 16kHz reference; outputs 48kHz via AudioVAE V2's built-in super-resolution, no external upsampler needed | |
| - π§ **Context-Aware Synthesis** β Automatically infers appropriate prosody and expressiveness from text content | |
| - β‘ **Real-Time Streaming** β RTF as low as ~0.3 on NVIDIA RTX 4090, and ~0.13 accelerated by [Nano-VLLM](https://github.com/a710128/nanovllm-voxcpm) | |
| - π **Fully Open-Source & Commercial-Ready** β Apache-2.0 license, free for commercial use | |
| <summary><b>Supported Languages (30)</b></summary> | |
| Arabic, Burmese, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, Tagalog, Thai, Turkish, Vietnamese | |
| Chinese Dialects: εε·θ―, η²€θ―, ε΄θ―, δΈεθ―, ζ²³εθ―, ιθ₯Ώθ―, ε±±δΈθ―, 倩ζ΄₯θ―, ι½εθ― | |
| ## Quick Start | |
| ### Installation | |
| ```bash | |
| pip install voxcpm | |
| ``` | |
| **Requirements:** Python β₯ 3.10, PyTorch β₯ 2.5.0, CUDA β₯ 12.0 Β· [Full Quick Start β](https://voxcpm.readthedocs.io/en/latest/quickstart.html) | |
| ### Text-to-Speech | |
| ```python | |
| from voxcpm import VoxCPM | |
| import soundfile as sf | |
| model = VoxCPM.from_pretrained("openbmb/VoxCPM2", load_denoiser=False) | |
| wav = model.generate( | |
| text="VoxCPM2 brings multilingual support, creative voice design, and controllable voice cloning.", | |
| cfg_value=2.0, | |
| inference_timesteps=10, | |
| ) | |
| sf.write("output.wav", wav, model.tts_model.sample_rate) | |
| ``` | |
| ### Voice Design | |
| Put the voice description in parentheses at the start of `text`, followed by the content to synthesize: | |
| ```python | |
| wav = model.generate( | |
| text="(A young woman, gentle and sweet voice)Hello, welcome to VoxCPM2!", | |
| cfg_value=2.0, | |
| inference_timesteps=10, | |
| ) | |
| sf.write("voice_design.wav", wav, model.tts_model.sample_rate) | |
| ``` | |
| ### Controllable Voice Cloning | |
| ```python | |
| # Basic cloning | |
| wav = model.generate( | |
| text="This is a cloned voice generated by VoxCPM2.", | |
| reference_wav_path="speaker.wav", | |
| ) | |
| sf.write("clone.wav", wav, model.tts_model.sample_rate) | |
| # Cloning with style control | |
| wav = model.generate( | |
| text="(slightly faster, cheerful tone)This is a cloned voice with style control.", | |
| reference_wav_path="speaker.wav", | |
| cfg_value=2.0, | |
| inference_timesteps=10, | |
| ) | |
| sf.write("controllable_clone.wav", wav, model.tts_model.sample_rate) | |
| ``` | |
| ### Ultimate Cloning | |
| Provide both the reference audio and its exact transcript for maximum fidelity. Pass the same clip to both `reference_wav_path` and `prompt_wav_path` for highest similarity: | |
| ```python | |
| wav = model.generate( | |
| text="This is an ultimate cloning demonstration using VoxCPM2.", | |
| prompt_wav_path="speaker_reference.wav", | |
| prompt_text="The transcript of the reference audio.", | |
| reference_wav_path="speaker_reference.wav", | |
| ) | |
| sf.write("hifi_clone.wav", wav, model.tts_model.sample_rate) | |
| ``` | |
| ### Streaming | |
| ```python | |
| import numpy as np | |
| chunks = [] | |
| for chunk in model.generate_streaming(text="Streaming is easy with VoxCPM!"): | |
| chunks.append(chunk) | |
| wav = np.concatenate(chunks) | |
| sf.write("streaming.wav", wav, model.tts_model.sample_rate) | |
| ``` | |
| ## Model Details | |
| | Property | Value | | |
| |---|---| | |
| | Architecture | Tokenizer-free Diffusion Autoregressive (LocEnc β TSLM β RALM β LocDiT) | | |
| | Backbone | Based on MiniCPM-4, totally 2B parameters | | |
| | Audio VAE | AudioVAE V2 (asymmetric encode/decode, 16kHz in β 48kHz out) | | |
| | Training Data | 2M+ hours multilingual speech | | |
| | LM Token Rate | 6.25 Hz | | |
| | Max Sequence Length | 8192 tokens | | |
| | dtype | bfloat16 | | |
| | VRAM | ~8 GB | | |
| | RTF (RTX 4090) | ~0.30 (standard) / ~0.13 (Nano-vLLM) | | |
| ## Performance | |
| VoxCPM2 achieves state-of-the-art or competitive results on major zero-shot and controllable TTS benchmarks. | |
| See the [GitHub repo](https://github.com/OpenBMB/VoxCPM#-performance) for full benchmark tables (Seed-TTS-eval, CV3-eval, InstructTTSEval, MiniMax Multilingual Test). | |
| ## Fine-tuning | |
| VoxCPM2 supports both full SFT and LoRA fine-tuning with as little as 5β10 minutes of audio: | |
| ```bash | |
| # LoRA fine-tuning (recommended) | |
| python scripts/train_voxcpm_finetune.py \ | |
| --config_path conf/voxcpm_v2/voxcpm_finetune_lora.yaml | |
| # Full fine-tuning | |
| python scripts/train_voxcpm_finetune.py \ | |
| --config_path conf/voxcpm_v2/voxcpm_finetune_all.yaml | |
| ``` | |
| See the [Fine-tuning Guide](https://voxcpm.readthedocs.io/en/latest/finetuning/finetune.html) for full instructions. | |
| ## Limitations | |
| - Voice Design and Style Control results may vary between runs; generating 1β3 times is recommended to obtain the desired output. | |
| - Performance varies across languages depending on training data availability. | |
| - Occasional instability may occur with very long or highly expressive inputs. | |
| - **Strictly forbidden** to use for impersonation, fraud, or disinformation. AI-generated content should be clearly labeled. | |
| ## Citation | |
| ```bibtex | |
| @article{voxcpm2_2026, | |
| title = {VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning}, | |
| author = {VoxCPM Team}, | |
| journal = {GitHub}, | |
| year = {2026}, | |
| } | |
| @article{voxcpm2025, | |
| title = {VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning}, | |
| author = {Zhou, Yixuan and Zeng, Guoyang and Liu, Xin and Li, Xiang and | |
| Yu, Renjie and Wang, Ziyang and Ye, Runchuan and Sun, Weiyue and | |
| Gui, Jiancheng and Li, Kehan and Wu, Zhiyong and Liu, Zhiyuan}, | |
| journal = {arXiv preprint arXiv:2509.24650}, | |
| year = {2025}, | |
| } | |
| ``` | |
| ## License | |
| Released under the [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) license, free for commercial use. For production deployments, we recommend thorough testing and safety evaluation tailored to your use case. | |