jennyxue commited on
Commit
df9a27d
Β·
verified Β·
1 Parent(s): 489784e

Model card and the original license

Browse files
Files changed (1) hide show
  1. README.md +277 -0
README.md ADDED
@@ -0,0 +1,277 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: openbmb/VoxCPM2
3
+ base_model_relation: quantized
4
+ license: apache-2.0
5
+ pipeline_tag: text-to-speech
6
+ library_name: mlx-audio
7
+ tags:
8
+ - mlx
9
+ - moonbox
10
+ - mlx-audio
11
+ - voxcpm
12
+ - voxcpm2
13
+ - text-to-speech
14
+ - voice-cloning
15
+ - speech
16
+ ---
17
+
18
+ # VoxCPM2 MLX
19
+
20
+ MLX builds of [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2), made by Moonbox for Apple Silicon Macs. Each precision is a folder; keep a folder's
21
+ files together. Only the parts listed below are converted: the rest of the pipeline comes from the original
22
+ repository unless the section below says otherwise.
23
+
24
+ ## Files
25
+
26
+ The last column is how far each build is from the original PyTorch model on the same inputs (see the section below
27
+ for what was measured).
28
+
29
+ | Precision | File | Size | sha256 | Checked against the original |
30
+ |---|---|---|---|---|
31
+ | bf16 | [`bf16/config.json`](https://huggingface.co/moonbox-dev/VoxCPM2-MLX/blob/main/bf16/config.json) | 6.6 kB | `0b5bee352207` | WER 0.0 % / CER 0.0 %, speaker sim 0.973 / 0.967, DNSMOS 3.12 / 2.33 (original 0.0 % / 0.0 %, 0.972 / 0.964, 2.93 / 2.28) |
32
+ | bf16 | [`bf16/model.safetensors`](https://huggingface.co/moonbox-dev/VoxCPM2-MLX/blob/main/bf16/model.safetensors) | 4.77 GB | `8ec57a6a9761` | WER 0.0 % / CER 0.0 %, speaker sim 0.973 / 0.967, DNSMOS 3.12 / 2.33 (original 0.0 % / 0.0 %, 0.972 / 0.964, 2.93 / 2.28) |
33
+ | bf16 | [`bf16/model.safetensors.index.json`](https://huggingface.co/moonbox-dev/VoxCPM2-MLX/blob/main/bf16/model.safetensors.index.json) | 65.7 kB | `1a4680fb582f` | WER 0.0 % / CER 0.0 %, speaker sim 0.973 / 0.967, DNSMOS 3.12 / 2.33 (original 0.0 % / 0.0 %, 0.972 / 0.964, 2.93 / 2.28) |
34
+ | bf16 | [`bf16/special_tokens_map.json`](https://huggingface.co/moonbox-dev/VoxCPM2-MLX/blob/main/bf16/special_tokens_map.json) | 1.6 kB | `068594063e37` | WER 0.0 % / CER 0.0 %, speaker sim 0.973 / 0.967, DNSMOS 3.12 / 2.33 (original 0.0 % / 0.0 %, 0.972 / 0.964, 2.93 / 2.28) |
35
+ | bf16 | [`bf16/tokenizer.json`](https://huggingface.co/moonbox-dev/VoxCPM2-MLX/blob/main/bf16/tokenizer.json) | 3.7 MB | `f8984687e4a9` | WER 0.0 % / CER 0.0 %, speaker sim 0.973 / 0.967, DNSMOS 3.12 / 2.33 (original 0.0 % / 0.0 %, 0.972 / 0.964, 2.93 / 2.28) |
36
+ | bf16 | [`bf16/tokenizer_config.json`](https://huggingface.co/moonbox-dev/VoxCPM2-MLX/blob/main/bf16/tokenizer_config.json) | 5.0 kB | `418be464e38f` | WER 0.0 % / CER 0.0 %, speaker sim 0.973 / 0.967, DNSMOS 3.12 / 2.33 (original 0.0 % / 0.0 %, 0.972 / 0.964, 2.93 / 2.28) |
37
+
38
+ ## How to run
39
+
40
+ VoxCPM2 (2B, tokenizer-free multilingual TTS with 48 kHz output: zero-shot speech, voice design and voice cloning)
41
+ for [mlx-audio](https://github.com/Blaizzy/mlx-audio). Each folder is a complete mlx-audio model: `model.safetensors`
42
+ (the MiniCPM backbone, residual LM, local encoder, flow-matching DiT and the AudioVAE), `config.json` and the
43
+ tokenizer. Tested with `mlx-audio==0.6.5`:
44
+
45
+ ```python
46
+ from mlx_audio.tts.utils import load_model
47
+ import numpy as np, soundfile as sf
48
+
49
+ model = load_model("8bit") # a downloaded folder
50
+ res = list(model.generate(text="Hello from Moonbox.",
51
+ ref_audio="voice.wav", # voice cloning
52
+ prompt_audio="voice.wav", prompt_text="Exact transcript of voice.wav."))
53
+ sf.write("out.wav", np.concatenate([np.array(r.audio) for r in res]), res[0].sample_rate) # 48 kHz
54
+ ```
55
+
56
+ Voice design: `model.generate(text=..., instruct="A young woman, warm and gentle voice")`. Defaults are the
57
+ original's: 10 flow-matching steps, CFG 2.0.
58
+
59
+ The tokenizer is the plain Llama tokenizer: mlx-audio splits multi-character Chinese tokens itself, which the
60
+ original repository does with remote code (`tokenization_voxcpm2.py`, left out here). `8bit/`, `6bit/` and `4bit/`
61
+ quantize the transformer layers (MLX affine, group size 64); the AudioVAE's convolutions stay bf16.
62
+
63
+ **How they were checked.** Voice cloning of 6 English and 4 Chinese sentences, two seeds each (the prompt both as
64
+ reference audio and as continuation prefix), with the original PyTorch model (float32, `voxcpm` 2.0.3, its bad-case
65
+ retry off) and every MLX build. Judged like the original: Whisper large-v3-turbo WER (English) / CER (Chinese),
66
+ WavLM-base-plus-sv speaker similarity to the prompt, DNSMOS P.835. Pass: error rate at most 2 points above the
67
+ original's, similarity at most 0.03 below, DNSMOS at most 0.2 below. The Chinese prompt is a noisy AISHELL clip,
68
+ which the model reproduces (hence the low Chinese DNSMOS for every system).
69
+
70
+ | Precision | Size | WER en / CER zh | Speaker similarity en / zh | DNSMOS en / zh | Real-time factor, M5 24 GB |
71
+ |---|---|---|---|---|---|
72
+ | original (PyTorch fp32, MPS) | - | 0.0 % / 0.0 % | 0.972 / 0.964 | 2.93 / 2.28 | 1.89 |
73
+ | `bf16/` | 4.77 GB | 0.0 % / 0.0 % | 0.973 / 0.967 | 3.12 / 2.33 | 1.08 |
74
+ | `fp16/` | 4.77 GB | 0.0 % / 0.0 % | 0.970 / 0.967 | 3.06 / 2.11 | 1.12 |
75
+ | `8bit/` | 2.62 GB | 0.0 % / 0.0 % | 0.975 / 0.966 | 3.12 / 2.31 | 1.29 |
76
+ | `6bit/` | 2.05 GB | 0.0 % / 0.0 % | 0.973 / 0.966 | 3.07 / 2.55 | 1.30 |
77
+ | `4bit/` | 1.48 GB | 0.0 % / 0.0 % | 0.972 / 0.964 | 3.12 / 2.22 | 1.14 |
78
+
79
+ Real-time factor = generation time / audio length (lower is faster), one sentence at a time.
80
+
81
+ ## How these were made
82
+
83
+ From [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) at revision `7936080b41`: the original weights loaded in MLX and stored as bf16 or fp16, or
84
+ quantized with MLX's affine quantization (`8bit`, `6bit`, `4bit`, group size 64; layers MLX can't quantize, such as
85
+ convolutions and norms, stay bf16). Every build was checked against the original before upload; the `sha256` column
86
+ above is what the files must hash to.
87
+
88
+ ## License
89
+
90
+ Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this
91
+ repository, and it governs your use of these files. The original model card follows unchanged.
92
+
93
+ ---
94
+
95
+ # Original model card
96
+
97
+ # VoxCPM2
98
+
99
+ **VoxCPM2** is a tokenizer-free, diffusion autoregressive Text-to-Speech model β€” **2B parameters**, **30 languages**, **48kHz** audio output, trained on over **2 million hours** of multilingual speech data.
100
+
101
+ [![GitHub](https://img.shields.io/badge/GitHub-VoxCPM-blue?logo=github)](https://github.com/OpenBMB/VoxCPM)
102
+ [![Docs](https://img.shields.io/badge/Docs-ReadTheDocs-8CA1AF)](https://voxcpm.readthedocs.io/en/latest/)
103
+ [![Demo](https://img.shields.io/badge/Live%20Playground-Demo-orange)](https://huggingface.co/spaces/OpenBMB/VoxCPM-Demo)
104
+ [![Audio Samples](https://img.shields.io/badge/Audio%20Samples-Demo%20Page-green)](https://openbmb.github.io/voxcpm2-demopage)
105
+ [![Discord](https://img.shields.io/badge/Discord-VoxCPM-5865F2?logo=discord&logoColor=white)](https://discord.gg/KZUx7tVNwz)
106
+ [![Lark](https://img.shields.io/badge/飞书羀-VoxCPM-00D6B9?logo=lark&logoColor=white)](https://applink.feishu.cn/client/chat/chatter/add_by_link?link_token=acds0b9d-23d8-4d7e-b696-d200f3e22a7f) [![MiniCPM Wiki](https://img.shields.io/badge/MiniCPM-Wiki-1677FF?logo=readthedocs&logoColor=white)](https://openbmb.cn/docs/)
107
+
108
+ ## Highlights
109
+
110
+ - 🌍 **30-Language Multilingual** β€” No language tag needed; input text in any supported language directly
111
+ - 🎨 **Voice Design** β€” Generate a novel voice from a natural-language description alone (gender, age, tone, emotion, pace…); no reference audio required
112
+ - πŸŽ›οΈ **Controllable Cloning** β€” Clone any voice from a short clip, with optional style guidance to steer emotion, pace, and expression while preserving timbre
113
+ - πŸŽ™οΈ **Ultimate Cloning** β€” Provide reference audio + its transcript for audio-continuation cloning; every vocal nuance faithfully reproduced
114
+ - πŸ”Š **48kHz Studio-Quality Output** β€” Accepts 16kHz reference; outputs 48kHz via AudioVAE V2's built-in super-resolution, no external upsampler needed
115
+ - 🧠 **Context-Aware Synthesis** β€” Automatically infers appropriate prosody and expressiveness from text content
116
+ - ⚑ **Real-Time Streaming** β€” RTF as low as ~0.3 on NVIDIA RTX 4090, and ~0.13 accelerated by [Nano-VLLM](https://github.com/a710128/nanovllm-voxcpm)
117
+ - πŸ“œ **Fully Open-Source & Commercial-Ready** β€” Apache-2.0 license, free for commercial use
118
+
119
+
120
+ <summary><b>Supported Languages (30)</b></summary>
121
+
122
+ Arabic, Burmese, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, Tagalog, Thai, Turkish, Vietnamese
123
+
124
+ Chinese Dialects: 四川话, η²€θ―­, 吴语, δΈœεŒ—θ―, 河南话, ι™•θ₯Ώθ―, 山东话, 倩ζ΄₯话, 闽南话
125
+
126
+
127
+ ## Quick Start
128
+
129
+ ### Installation
130
+
131
+ ```bash
132
+ pip install voxcpm
133
+ ```
134
+
135
+ **Requirements:** Python β‰₯ 3.10, PyTorch β‰₯ 2.5.0, CUDA β‰₯ 12.0 Β· [Full Quick Start β†’](https://voxcpm.readthedocs.io/en/latest/quickstart.html)
136
+
137
+ ### Text-to-Speech
138
+
139
+ ```python
140
+ from voxcpm import VoxCPM
141
+ import soundfile as sf
142
+
143
+ model = VoxCPM.from_pretrained("openbmb/VoxCPM2", load_denoiser=False)
144
+
145
+ wav = model.generate(
146
+ text="VoxCPM2 brings multilingual support, creative voice design, and controllable voice cloning.",
147
+ cfg_value=2.0,
148
+ inference_timesteps=10,
149
+ )
150
+ sf.write("output.wav", wav, model.tts_model.sample_rate)
151
+ ```
152
+
153
+ ### Voice Design
154
+
155
+ Put the voice description in parentheses at the start of `text`, followed by the content to synthesize:
156
+
157
+ ```python
158
+ wav = model.generate(
159
+ text="(A young woman, gentle and sweet voice)Hello, welcome to VoxCPM2!",
160
+ cfg_value=2.0,
161
+ inference_timesteps=10,
162
+ )
163
+ sf.write("voice_design.wav", wav, model.tts_model.sample_rate)
164
+ ```
165
+
166
+ ### Controllable Voice Cloning
167
+
168
+ ```python
169
+ # Basic cloning
170
+ wav = model.generate(
171
+ text="This is a cloned voice generated by VoxCPM2.",
172
+ reference_wav_path="speaker.wav",
173
+ )
174
+ sf.write("clone.wav", wav, model.tts_model.sample_rate)
175
+
176
+ # Cloning with style control
177
+ wav = model.generate(
178
+ text="(slightly faster, cheerful tone)This is a cloned voice with style control.",
179
+ reference_wav_path="speaker.wav",
180
+ cfg_value=2.0,
181
+ inference_timesteps=10,
182
+ )
183
+ sf.write("controllable_clone.wav", wav, model.tts_model.sample_rate)
184
+ ```
185
+
186
+ ### Ultimate Cloning
187
+
188
+ Provide both the reference audio and its exact transcript for maximum fidelity. Pass the same clip to both `reference_wav_path` and `prompt_wav_path` for highest similarity:
189
+
190
+ ```python
191
+ wav = model.generate(
192
+ text="This is an ultimate cloning demonstration using VoxCPM2.",
193
+ prompt_wav_path="speaker_reference.wav",
194
+ prompt_text="The transcript of the reference audio.",
195
+ reference_wav_path="speaker_reference.wav",
196
+ )
197
+ sf.write("hifi_clone.wav", wav, model.tts_model.sample_rate)
198
+ ```
199
+
200
+ ### Streaming
201
+
202
+ ```python
203
+ import numpy as np
204
+
205
+ chunks = []
206
+ for chunk in model.generate_streaming(text="Streaming is easy with VoxCPM!"):
207
+ chunks.append(chunk)
208
+ wav = np.concatenate(chunks)
209
+ sf.write("streaming.wav", wav, model.tts_model.sample_rate)
210
+ ```
211
+
212
+ ## Model Details
213
+
214
+ | Property | Value |
215
+ |---|---|
216
+ | Architecture | Tokenizer-free Diffusion Autoregressive (LocEnc β†’ TSLM β†’ RALM β†’ LocDiT) |
217
+ | Backbone | Based on MiniCPM-4, totally 2B parameters |
218
+ | Audio VAE | AudioVAE V2 (asymmetric encode/decode, 16kHz in β†’ 48kHz out) |
219
+ | Training Data | 2M+ hours multilingual speech |
220
+ | LM Token Rate | 6.25 Hz |
221
+ | Max Sequence Length | 8192 tokens |
222
+ | dtype | bfloat16 |
223
+ | VRAM | ~8 GB |
224
+ | RTF (RTX 4090) | ~0.30 (standard) / ~0.13 (Nano-vLLM) |
225
+
226
+ ## Performance
227
+
228
+ VoxCPM2 achieves state-of-the-art or competitive results on major zero-shot and controllable TTS benchmarks.
229
+
230
+ See the [GitHub repo](https://github.com/OpenBMB/VoxCPM#-performance) for full benchmark tables (Seed-TTS-eval, CV3-eval, InstructTTSEval, MiniMax Multilingual Test).
231
+
232
+ ## Fine-tuning
233
+
234
+ VoxCPM2 supports both full SFT and LoRA fine-tuning with as little as 5–10 minutes of audio:
235
+
236
+ ```bash
237
+ # LoRA fine-tuning (recommended)
238
+ python scripts/train_voxcpm_finetune.py \
239
+ --config_path conf/voxcpm_v2/voxcpm_finetune_lora.yaml
240
+
241
+ # Full fine-tuning
242
+ python scripts/train_voxcpm_finetune.py \
243
+ --config_path conf/voxcpm_v2/voxcpm_finetune_all.yaml
244
+ ```
245
+
246
+ See the [Fine-tuning Guide](https://voxcpm.readthedocs.io/en/latest/finetuning/finetune.html) for full instructions.
247
+
248
+ ## Limitations
249
+
250
+ - Voice Design and Style Control results may vary between runs; generating 1–3 times is recommended to obtain the desired output.
251
+ - Performance varies across languages depending on training data availability.
252
+ - Occasional instability may occur with very long or highly expressive inputs.
253
+ - **Strictly forbidden** to use for impersonation, fraud, or disinformation. AI-generated content should be clearly labeled.
254
+
255
+ ## Citation
256
+
257
+ ```bibtex
258
+ @article{voxcpm2_2026,
259
+ title = {VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning},
260
+ author = {VoxCPM Team},
261
+ journal = {GitHub},
262
+ year = {2026},
263
+ }
264
+
265
+ @article{voxcpm2025,
266
+ title = {VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning},
267
+ author = {Zhou, Yixuan and Zeng, Guoyang and Liu, Xin and Li, Xiang and
268
+ Yu, Renjie and Wang, Ziyang and Ye, Runchuan and Sun, Weiyue and
269
+ Gui, Jiancheng and Li, Kehan and Wu, Zhiyong and Liu, Zhiyuan},
270
+ journal = {arXiv preprint arXiv:2509.24650},
271
+ year = {2025},
272
+ }
273
+ ```
274
+
275
+ ## License
276
+
277
+ Released under the [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) license, free for commercial use. For production deployments, we recommend thorough testing and safety evaluation tailored to your use case.