VoxCPM2 ONNX

ONNX builds of VoxCPM2, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder holding the graphs and a small Python pipeline.py that does the rest (features, prompt, sampling loop); what the graphs take and return is under How to run.

Files

The last column is the benchmark on Moonbox's TTS set (voice cloning of English and Chinese sentences from a LibriSpeech and an AISHELL-1 prompt; details under How to run): error rate of a Whisper transcript against the text, speaker similarity to the prompt and DNSMOS, against the original PyTorch model.

Precision File Size sha256 Benchmark
fp16 fp16/base_lm.onnx 17.3 MB 3a5d89c65ba3 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/base_lm.onnx.data 2.88 GB 42ac51061af3 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/estimator.onnx 424.6 MB 057b45e894b4 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/feat_encoder.onnx 436.9 MB 38182a733bf8 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/fsq.onnx 4.2 MB 78076e951846 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/fusion.onnx 16.8 MB 39916677f91b WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/heads.onnx 16.8 MB 19d9f4f6d250 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/multichar_tokens.json 0.2 MB 43263698b2c5 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/pipeline.py 9.5 kB 6bee96bae464 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/residual_lm.onnx 0.1 MB 55248bc9d387 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/residual_lm.onnx.data 822.2 MB 23082e0ccbff WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/text_embed.onnx 300.8 MB 48be9c770f79 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/tokenizer.json 3.7 MB f8984687e4a9 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/vae_decoder.onnx 91.8 MB e3205805aa8d WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
fp16 fp16/vae_encoder.onnx 96.3 MB f912268155d7 WER 0.3 % / CER 0.0 % (original 0.0 / 0.0, bf16), similarity 0.973 / 0.966 (0.976 / 0.965), DNSMOS 3.06 / 2.10 (3.07 / 2.29); RTF 0.85 (CUDA)
int8 int8/base_lm.onnx 1.43 GB e498cb38af6d WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/estimator.onnx 213.5 MB 890a11ed3561 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/feat_encoder.onnx 244.7 MB 0d3b35cac7f3 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/fsq.onnx 2.1 MB 0e53a24ebbed WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/fusion.onnx 8.4 MB 2ce552d569a2 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/heads.onnx 8.4 MB 9a60167d9bac WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/multichar_tokens.json 0.2 MB 43263698b2c5 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/pipeline.py 9.5 kB 6bee96bae464 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/residual_lm.onnx 403.4 MB e557865390b8 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/text_embed.onnx 150.7 MB 06477af2a9f1 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/tokenizer.json 3.7 MB f8984687e4a9 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/vae_decoder.onnx 46.3 MB 126eba421426 WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int8 int8/vae_encoder.onnx 48.4 MB b5b89a6b44be WER 0.0 % / CER 0.3 %, similarity 0.973 / 0.968, DNSMOS 2.99 / 2.15; RTF 1.7 (CUDA)
int4 int4/base_lm.onnx 768.1 MB 40c52af0269d WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/estimator.onnx 213.5 MB 890a11ed3561 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/feat_encoder.onnx 244.7 MB 0d3b35cac7f3 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/fsq.onnx 2.1 MB 0e53a24ebbed WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/fusion.onnx 8.4 MB 2ce552d569a2 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/heads.onnx 8.4 MB 9a60167d9bac WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/multichar_tokens.json 0.2 MB 43263698b2c5 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/pipeline.py 9.5 kB 6bee96bae464 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/residual_lm.onnx 214.6 MB 771cdf95aaf8 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/text_embed.onnx 150.7 MB 06477af2a9f1 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/tokenizer.json 3.7 MB f8984687e4a9 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/vae_decoder.onnx 46.3 MB 126eba421426 WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU
int4 int4/vae_encoder.onnx 48.4 MB b5b89a6b44be WER 0.3 % / CER 0.0 %, similarity 0.976 / 0.969, DNSMOS 3.01 / 2.11; RTF 1.7 (CUDA), about 33 on a laptop CPU

How to run

VoxCPM2 (OpenBMB's tokenizer-free speech generator: voice cloning from a reference recording and its transcript, English and Chinese, 48 kHz output) as ten ONNX graphs plus pipeline.py, a numpy runtime that reproduces voxcpm 2.0.3's VoxCPM.generate(text, prompt_wav_path, prompt_text, reference_wav_path) around them:

  • base_lm.onnx / residual_lm.onnx: the two MiniCPM4 language models (28 and 8 layers), with the KV cache as inputs and outputs: inputs_embeds [1, S, 2048], past_key / past_value [layers, 1, 2, P, 128] -> the final normed hidden [1, S, 2048] and present_key / present_value [layers, 1, 2, P + S, 128].
  • estimator.onnx: the local diffusion transformer that turns each LM step into one patch of 4 latent frames. x, cond [2, 64, 4], mu [2, 2048], t, dt [2] -> velocity [2, 64, 4] (conditional and unconditional rows, for classifier-free guidance).
  • feat_encoder.onnx (latent patches -> LM inputs), text_embed.onnx, fsq.onnx (the scalar quantizer on the LM output), fusion.onnx (LM output + audio embedding -> residual LM input), heads.onnx (-> the DiT condition and the stop logits).
  • vae_encoder.onnx / vae_decoder.onnx: the audio VAE. 16 kHz audio [1, 1, N] -> latents [1, 64, N / 640], and latents [1, 64, T] -> 48 kHz audio [1, 1, 1920 T].

pipeline.py builds the original's prompt (the reference audio as a reference prefix and as the prompt the speech continues from, then the prompt transcript and the text), runs the LM / DiT loop with 10 Euler steps, classifier-free guidance 2.0 (CFG-zero*) and the original's length limit (6 x the text's tokens + 10 patches), and decodes. It needs numpy, onnxruntime (or onnxruntime-gpu), tokenizers, soundfile and soxr:

python fp16/pipeline.py "The quick brown fox jumps over the lazy dog." out.wav --ref voice.wav \
    --ref-text "Exactly what is said in voice.wav." --cuda --seed 0
import sys, soundfile as sf
sys.path.insert(0, "fp16")
import pipeline
tts = pipeline.VoxCPM2("fp16", "cuda")
ref, sr = sf.read("voice.wav", dtype="float32")
wav, out_sr = tts("Text to say.", ref, sr, "Transcript of voice.wav.", seed=0)      # 48 kHz

The diffusion noise comes from numpy's generator, so a seed gives the same speech every run here, not the same speech as PyTorch with that seed. The original's text normalisation and its retry on overlong output are off by default and not included; pass normalised text.

Precisions. fp16/: both language models in fp16, every other graph fp16 weights with fp32 math (the diffusion transformer, the local encoder and the VAE run in fp32 in the original). int8/: the language models 8-bit MatMulNBits (block 32), every other graph weight-only int8, the text embedding int8 per row. int4/: the language models 4-bit MatMulNBits (block 32), the rest as int8/. No bf16 build. These are GPU builds: int4/ on a laptop CPU runs about 33 times slower than real time (measured on 7 clips).

Benchmark. The lab's TTS set: 6 English sentences cloned from a LibriSpeech test-clean prompt and 4 Chinese sentences from an AISHELL-1 prompt, each with seeds 0-3 (40 clips per build), the original's settings (prompt = reference = the set's prompt, CFG 2.0, 10 steps, no retry). Sampled speech can't be compared sample by sample, so each build is judged as the original is: Whisper large-v3-turbo's WER (English) or CER (Chinese) against the text, WavLM-base-plus-sv speaker similarity to the prompt, and DNSMOS (P.835 overall). The original is voxcpm 2.0.3 in bf16 (the checkpoint's dtype) on CUDA. Pass (the TTS threshold used across these repositories): per language, error rate at most 2 points above the original's, similarity at most 0.03 below, DNSMOS at most 0.2 below. Real-time factor (processing time / audio time) on an RTX 3080 Laptop GPU, whole pipeline.

English WER similarity DNSMOS Chinese CER similarity DNSMOS Real-time factor
original (PyTorch bf16, CUDA) 0.0 % 0.976 3.07 0.0 % 0.965 2.29 2.2
fp16/ (CUDA) 0.3 % 0.973 3.06 0.0 % 0.966 2.10 0.85
int8/ (CUDA) 0.0 % 0.973 2.99 0.3 % 0.968 2.15 1.7
int4/ (CUDA) 0.3 % 0.976 3.01 0.0 % 0.969 2.11 1.7

DNSMOS of this model's output varies a lot from sample to sample: the original itself scored 3.17 / 2.36 on seeds 0-1 and 3.07 / 2.29 on seeds 0-3 here (and 2.93 / 2.28 in fp32 on another machine), so the margins above are within that spread rather than measured differences between builds.

How these were made

From VoxCPM2 at revision 7936080b41. The graphs were exported from the original PyTorch modules with torch.onnx (opset 17 / 18) and checked against them; language-model decoders were built with onnxruntime-genai's model builder (inputs_embeds in, KV cache in and out). Weight-only quantization: int8 is a per-output-channel int8 weight behind DequantizeLinear (or 8-bit MatMulNBits in a text decoder), int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was benchmarked against the original before upload (pass, per language: error rate at most 2 points above the original's, speaker similarity at most 0.03 below, DNSMOS at most 0.2 below), and the sha256 column above is what the files must hash to.

License

Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.


Original model card

VoxCPM2

VoxCPM2 is a tokenizer-free, diffusion autoregressive Text-to-Speech model β€” 2B parameters, 30 languages, 48kHz audio output, trained on over 2 million hours of multilingual speech data.

GitHub Docs Demo Audio Samples Discord Lark MiniCPM Wiki

Highlights

  • 🌍 30-Language Multilingual β€” No language tag needed; input text in any supported language directly
  • 🎨 Voice Design β€” Generate a novel voice from a natural-language description alone (gender, age, tone, emotion, pace…); no reference audio required
  • πŸŽ›οΈ Controllable Cloning β€” Clone any voice from a short clip, with optional style guidance to steer emotion, pace, and expression while preserving timbre
  • πŸŽ™οΈ Ultimate Cloning β€” Provide reference audio + its transcript for audio-continuation cloning; every vocal nuance faithfully reproduced
  • πŸ”Š 48kHz Studio-Quality Output β€” Accepts 16kHz reference; outputs 48kHz via AudioVAE V2's built-in super-resolution, no external upsampler needed
  • 🧠 Context-Aware Synthesis β€” Automatically infers appropriate prosody and expressiveness from text content
  • ⚑ Real-Time Streaming β€” RTF as low as ~0.3 on NVIDIA RTX 4090, and ~0.13 accelerated by Nano-VLLM
  • πŸ“œ Fully Open-Source & Commercial-Ready β€” Apache-2.0 license, free for commercial use
Supported Languages (30)

Arabic, Burmese, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, Tagalog, Thai, Turkish, Vietnamese

Chinese Dialects: 四川话, η²€θ―­, 吴语, δΈœεŒ—θ―, 河南话, ι™•θ₯Ώθ―, 山东话, 倩ζ΄₯话, 闽南话

Quick Start

Installation

pip install voxcpm

Requirements: Python β‰₯ 3.10, PyTorch β‰₯ 2.5.0, CUDA β‰₯ 12.0 Β· Full Quick Start β†’

Text-to-Speech

from voxcpm import VoxCPM
import soundfile as sf

model = VoxCPM.from_pretrained("openbmb/VoxCPM2", load_denoiser=False)

wav = model.generate(
    text="VoxCPM2 brings multilingual support, creative voice design, and controllable voice cloning.",
    cfg_value=2.0,
    inference_timesteps=10,
)
sf.write("output.wav", wav, model.tts_model.sample_rate)

Voice Design

Put the voice description in parentheses at the start of text, followed by the content to synthesize:

wav = model.generate(
    text="(A young woman, gentle and sweet voice)Hello, welcome to VoxCPM2!",
    cfg_value=2.0,
    inference_timesteps=10,
)
sf.write("voice_design.wav", wav, model.tts_model.sample_rate)

Controllable Voice Cloning

# Basic cloning
wav = model.generate(
    text="This is a cloned voice generated by VoxCPM2.",
    reference_wav_path="speaker.wav",
)
sf.write("clone.wav", wav, model.tts_model.sample_rate)

# Cloning with style control
wav = model.generate(
    text="(slightly faster, cheerful tone)This is a cloned voice with style control.",
    reference_wav_path="speaker.wav",
    cfg_value=2.0,
    inference_timesteps=10,
)
sf.write("controllable_clone.wav", wav, model.tts_model.sample_rate)

Ultimate Cloning

Provide both the reference audio and its exact transcript for maximum fidelity. Pass the same clip to both reference_wav_path and prompt_wav_path for highest similarity:

wav = model.generate(
    text="This is an ultimate cloning demonstration using VoxCPM2.",
    prompt_wav_path="speaker_reference.wav",
    prompt_text="The transcript of the reference audio.",
    reference_wav_path="speaker_reference.wav",
)
sf.write("hifi_clone.wav", wav, model.tts_model.sample_rate)

Streaming

import numpy as np

chunks = []
for chunk in model.generate_streaming(text="Streaming is easy with VoxCPM!"):
    chunks.append(chunk)
wav = np.concatenate(chunks)
sf.write("streaming.wav", wav, model.tts_model.sample_rate)

Model Details

Property Value
Architecture Tokenizer-free Diffusion Autoregressive (LocEnc β†’ TSLM β†’ RALM β†’ LocDiT)
Backbone Based on MiniCPM-4, totally 2B parameters
Audio VAE AudioVAE V2 (asymmetric encode/decode, 16kHz in β†’ 48kHz out)
Training Data 2M+ hours multilingual speech
LM Token Rate 6.25 Hz
Max Sequence Length 8192 tokens
dtype bfloat16
VRAM ~8 GB
RTF (RTX 4090) ~0.30 (standard) / ~0.13 (Nano-vLLM)

Performance

VoxCPM2 achieves state-of-the-art or competitive results on major zero-shot and controllable TTS benchmarks.

See the GitHub repo for full benchmark tables (Seed-TTS-eval, CV3-eval, InstructTTSEval, MiniMax Multilingual Test).

Fine-tuning

VoxCPM2 supports both full SFT and LoRA fine-tuning with as little as 5–10 minutes of audio:

# LoRA fine-tuning (recommended)
python scripts/train_voxcpm_finetune.py \
    --config_path conf/voxcpm_v2/voxcpm_finetune_lora.yaml

# Full fine-tuning
python scripts/train_voxcpm_finetune.py \
    --config_path conf/voxcpm_v2/voxcpm_finetune_all.yaml

See the Fine-tuning Guide for full instructions.

Limitations

  • Voice Design and Style Control results may vary between runs; generating 1–3 times is recommended to obtain the desired output.
  • Performance varies across languages depending on training data availability.
  • Occasional instability may occur with very long or highly expressive inputs.
  • Strictly forbidden to use for impersonation, fraud, or disinformation. AI-generated content should be clearly labeled.

Citation

@article{voxcpm2_2026,
  title   = {VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning},
  author  = {VoxCPM Team},
  journal = {GitHub},
  year    = {2026},
}

@article{voxcpm2025,
  title   = {VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning},
  author  = {Zhou, Yixuan and Zeng, Guoyang and Liu, Xin and Li, Xiang and
             Yu, Renjie and Wang, Ziyang and Ye, Runchuan and Sun, Weiyue and
             Gui, Jiancheng and Li, Kehan and Wu, Zhiyong and Liu, Zhiyuan},
  journal = {arXiv preprint arXiv:2509.24650},
  year    = {2025},
}

License

Released under the Apache-2.0 license, free for commercial use. For production deployments, we recommend thorough testing and safety evaluation tailored to your use case.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for moonbox-dev/VoxCPM2-ONNX

Base model

openbmb/VoxCPM2
Quantized
(29)
this model

Paper for moonbox-dev/VoxCPM2-ONNX