File size: 17,802 Bytes
dfb7eaf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
---
base_model: amphion/Vevo
base_model_relation: quantized
license: cc-by-nc-4.0
pipeline_tag: audio-to-audio
library_name: onnx
tags:
- onnx
- moonbox
- onnxruntime
- vevo
- amphion
- voice-conversion
- zero-shot
- non-commercial
---

# Vevo ONNX

ONNX builds of [Vevo](https://huggingface.co/amphion/Vevo), made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each
precision is one folder holding the graphs and a small Python `pipeline.py` that does the rest (features,
resampling, the conversion steps); what the graphs take and return is under *How to run*.

## Files

The last column is the benchmark on Moonbox's voice-conversion set (English LibriSpeech and Chinese AISHELL-1
recordings, each converted into an English and a Chinese target voice; details under *How to run*): error rate of a
Whisper transcript of the output against the source's transcript, speaker similarity to the target voice and DNSMOS,
against the original PyTorch model.

| Precision | File | Size | sha256 | Benchmark |
|---|---|---|---|---|
| int8 | [`int8/cond.onnx`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/cond.onnx) | 33.6 MB | `25c2af7d480f` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) |
| int8 | [`int8/config.json`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/config.json) | 0.0 kB | `21abd1bc2f85` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) |
| int8 | [`int8/content.onnx`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/content.onnx) | 0.3 MB | `2496a23dcfbc` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) |
| int8 | [`int8/content.onnx.data`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/content.onnx.data) | 1.05 GB | `68c51fe1ac74` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) |
| int8 | [`int8/estimator.onnx`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/estimator.onnx) | 331.2 MB | `aab0060de5a5` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) |
| int8 | [`int8/mel_basis.npy`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/mel_basis.npy) | 0.5 MB | `7bb7e7de4592` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) |
| int8 | [`int8/pipeline.py`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/pipeline.py) | 7.2 kB | `b87a739f78be` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) |
| int8 | [`int8/vocoder.onnx`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/vocoder.onnx) | 257.0 MB | `995abcdb8559` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) |

## How to run

Vevo-Timbre (Amphion's Vevo: zero-shot voice conversion that re-voices a recording into the timbre of a few seconds of
reference audio, keeping its words, timing and speaking style; 24 kHz output) as four ONNX graphs plus `pipeline.py`,
a numpy runtime that reproduces the official `VevoInferencePipeline.inference_fm` (`infer_vevotimbre.py`: 32
flow-matching steps, classifier-free guidance 1.0 rescaled 0.75) around them. Vevo's autoregressive models
(Vevo-Voice and Vevo-Style, which also convert accent and emotion, and Vevo-TTS) are not included.

- `content.onnx`: 16 kHz audio `wave` [1, N] -> content-style tokens `tokens` [1, N / 320] (50 Hz, 8192 codes:
  torchaudio's HuBERT-large layers 1-18, mean / std normalisation, the RepCodec encoder and vector quantizer).
- `cond.onnx`: tokens [1, L] -> the flow's condition `cond` [1, L, 1024].
- `estimator.onnx`: the flow-matching transformer: `x` [B, L, 128] (reference mel, then the noisy target), `t` [B],
  `cond` [B, L, 1024] -> `flow` [B, L, 128].
- `vocoder.onnx`: the Vocos vocoder up to its ISTFT head: normalised mel `mel` [1, 128, T] -> STFT `real`, `imag`
  [1, 961, T]. `pipeline.py` does the inverse STFT (n_fft 1920, hop 480, "same" padding) in numpy.

`pipeline.py` also does the resampling (soxr to 24 kHz, then torchaudio's sinc resampler ported to numpy for the 16
kHz tokenizer input, as the official `load_wav` does) and the 128-bin log-mel of the reference (`mel_basis.npy`,
normalised with `config.json`). The flow's starting noise comes from numpy's generator, so a seed gives the same
output every run here (not the same as PyTorch with that seed). The flow attends over reference and source together,
with no chunking: keep the reference to a few seconds and the source under about 30 s, as the official examples do.
It needs numpy, onnxruntime (or onnxruntime-gpu), soundfile and soxr:

```bash
python int8/pipeline.py source.wav voice.wav out.wav --cuda --seed 0
```

```python
import sys, soundfile as sf
sys.path.insert(0, "int8")
import pipeline
vc = pipeline.VevoTimbre("int8", "cuda")
src, sr = sf.read("source.wav", dtype="float32")
ref, ref_sr = sf.read("voice.wav", dtype="float32")
wav, out_sr = vc(src, sr, ref, ref_sr, steps=32, seed=0)       # 24 kHz
```

**Precisions.** `int8/` only (1.6 GB): weight-only int8 for the flow transformer, the vocoder and the condition
embedding, the content tokenizer in fp32 (1.26 GB of it). An fp16 tokenizer flips enough of its 8192 codes to put
Chinese CER at 22.9 % against the original's 4.2 %. With the tokenizer in fp32, fp16 math in the flow and vocoder still
gave 8.3 %, and a 4-bit flow transformer gave 14.6 %, so there is no `fp16/` or `int4/`. No bf16 build.

**Benchmark.** The lab's voice-conversion set: 4 English recordings (LibriSpeech test-clean) and 2 Chinese ones
(AISHELL-1 test), each converted into the English (LibriSpeech) and the Chinese (AISHELL-1) voice of the lab's TTS
prompts: 12 conversions. Each output is judged by Whisper large-v3-turbo's WER (English sources) or CER (Chinese)
against the source's transcript, WavLM-base-plus-sv speaker similarity to the target voice, and DNSMOS (P.835
overall). The original is the official code (Amphion @ 26f6883, `inference_fm`) in fp32 on CUDA, given the same
starting noise. Pass, per source language: error rate at most 2 points above the original's, similarity at most
0.03 below, DNSMOS at most 0.2 below. Real-time factor (processing time / audio time) on an RTX 3080 Laptop GPU,
whole pipeline. SNR is each output against the original's on the same input and noise.

| | English WER | similarity | DNSMOS | Chinese CER | similarity | DNSMOS | SNR vs original | Real-time factor |
|---|---|---|---|---|---|---|---|---|
| original (PyTorch fp32, CUDA) | 1.1 % | 0.907 | 2.92 | 4.2 % | 0.893 | 2.93 | | 0.54 |
| `int8/` (CUDA) | 1.1 % | 0.908 | 2.91 | 4.2 % | 0.899 | 2.90 | 8 dB | 0.44 |

The Chinese set is small (4 clips, about 50 characters: one character is 2 points). The graphs match the PyTorch
modules on test inputs (content tokens identical, the flow transformer 4e-6, vocoder with the numpy inverse STFT 9e-8);
the fp32 graphs reach 11 dB SNR against the original on this set, where 32 guided flow steps carry small differences
into the waveform.

**License.** The Vevo weights are CC-BY-NC-4.0 (amphion/Vevo): these files are too, so they may not be used
commercially; credit Amphion's Vevo (Zhang et al., "Vevo: Controllable Zero-Shot Voice Imitation with
Self-Supervised Disentanglement", ICLR 2025). The Amphion code they were exported from is MIT. The tokenizer contains
HuBERT-large from torchaudio's HUBERT_LARGE bundle (fairseq, MIT).

## How these were made

From [Vevo](https://huggingface.co/amphion/Vevo) at revision `7edf4640c4`. The graphs were exported from the original PyTorch modules with
torch.onnx (opset 17 / 18) and checked against them. Weight-only quantization: int8 is a per-output-channel int8
weight behind DequantizeLinear, int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was
benchmarked against the original before upload (pass, per language: error rate at most 2 points above the
original's, similarity to the target voice at most 0.03 below, DNSMOS at most 0.2 below), and the `sha256` column above is what
the files must hash to.

## License

CC-BY-NC-4.0. These files are a derivative of the original model and carry its license: it is copied in this
repository, and it governs your use of these files. The original model card follows unchanged.

---

# Original model card

# Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

[![arXiv](https://img.shields.io/badge/OpenReview-Paper-COLOR.svg)](https://openreview.net/pdf?id=anQDiQZhDP)
[![hf](https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-model-yellow)](https://huggingface.co/amphion/Vevo)
[![WebPage](https://img.shields.io/badge/WebPage-Demo-red)](https://versavoice.github.io/) [![readme](https://img.shields.io/badge/README-GitHub-blue)](https://github.com/open-mmlab/Amphion/blob/main/models/vc/vevo/README.md)

We present our reproduction of [Vevo](https://openreview.net/pdf?id=anQDiQZhDP), a versatile zero-shot voice imitation framework with controllable timbre and style. We invite you to explore the [audio samples](https://versavoice.github.io/) to experience Vevo's capabilities firsthand.

We have included the following pre-trained Vevo models at Amphion:

- **Vevo-Timbre**: It can conduct *style-preserved* voice conversion.
- **Vevo-Style**: It can conduct style conversion, such as *accent conversion* and *emotion conversion*.
- **Vevo-Voice**: It can conduct *style-converted* voice conversion.
- **Vevo-TTS**: It can conduct *style and timbre controllable* TTS.

Besides, we also release the **content tokenizer** and **content-style tokenizer** proposed by Vevo. Notably, all these pre-trained models are trained on [Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset), containing 101k hours of speech data among six languages (English, Chinese, German, French, Japanese, and Korean).

## Model Introduction

We provide the following pre-trained models:


| Model Name          | Description   |    
|-------------------|-------------|
| [Content Tokenizer](https://huggingface.co/amphion/Vevo/tree/main/tokenizer/vq32)      | Converting speech to content tokens. It is a single codebook VQ-VAE with a vocabulary size of 32. The frame rate is 50Hz.|
| [Content-Style Tokenizer](https://huggingface.co/amphion/Vevo/tree/main/tokenizer/vq8192)      | Converting speech to content-style tokens. It is a single codebook VQ-VAE with a vocabulary size of 8192. The frame rate is 50Hz.|
| [Vq32ToVq8192](https://huggingface.co/amphion/Vevo/tree/main/contentstyle_modeling/Vq32ToVq8192)      | Predicting content-style tokens from content tokens with an auto-regressive transformer (480M). |
| [PhoneToVq8192](https://huggingface.co/amphion/Vevo/tree/main/contentstyle_modeling/PhoneToVq8192)      | Predicting content-style tokens from phone tokens with an auto-regressive transformer (740M). |
| [Vq8192ToMels](https://huggingface.co/amphion/Vevo/tree/main/acoustic_modeling/Vq8192ToMels)      | Predicting mel-spectrogram from content-style tokens with a flow-matching transformer (330M). |
| [Vocoder](https://huggingface.co/amphion/Vevo/tree/main/acoustic_modeling/Vocoder)      | Predicting audio from mel-spectrogram with a Vocos-based vocoder (250M). |

You can download all pretrained checkpoints from [HuggingFace](https://huggingface.co/amphion/MaskGCT/tree/main) or use huggingface API.

## Usage

You can refer to our [recipe](https://github.com/open-mmlab/Amphion/blob/main/models/vc/vevo/README.md) at GitHub for more usage details. For example, to use Vevo-TTS, after you clone the Amphion github repository, you can use the script like:

```python
import os
from huggingface_hub import snapshot_download

from models.vc.vevo.vevo_utils import *


def vevo_tts(
    src_text,
    ref_wav_path,
    timbre_ref_wav_path=None,
    output_path=None,
    ref_text=None,
    src_language="en",
    ref_language="en",
):
    if timbre_ref_wav_path is None:
        timbre_ref_wav_path = ref_wav_path

    gen_audio = inference_pipeline.inference_ar_and_fm(
        src_wav_path=None,
        src_text=src_text,
        style_ref_wav_path=ref_wav_path,
        timbre_ref_wav_path=timbre_ref_wav_path,
        style_ref_wav_text=ref_text,
        src_text_language=src_language,
        style_ref_wav_text_language=ref_language,
    )

    assert output_path is not None
    save_audio(gen_audio, output_path=output_path)


if __name__ == "__main__":
    # ===== Device =====
    device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

    # ===== Content-Style Tokenizer =====
    local_dir = snapshot_download(
        repo_id="amphion/Vevo",
        repo_type="model",
        cache_dir="./ckpts/Vevo",
        allow_patterns=["tokenizer/vq8192/*"],
    )

    content_style_tokenizer_ckpt_path = os.path.join(local_dir, "tokenizer/vq8192")

    # ===== Autoregressive Transformer =====
    local_dir = snapshot_download(
        repo_id="amphion/Vevo",
        repo_type="model",
        cache_dir="./ckpts/Vevo",
        allow_patterns=["contentstyle_modeling/PhoneToVq8192/*"],
    )

    ar_cfg_path = "./models/vc/vevo/config/PhoneToVq8192.json"
    ar_ckpt_path = os.path.join(local_dir, "contentstyle_modeling/PhoneToVq8192")

    # ===== Flow Matching Transformer =====
    local_dir = snapshot_download(
        repo_id="amphion/Vevo",
        repo_type="model",
        cache_dir="./ckpts/Vevo",
        allow_patterns=["acoustic_modeling/Vq8192ToMels/*"],
    )

    fmt_cfg_path = "./models/vc/vevo/config/Vq8192ToMels.json"
    fmt_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vq8192ToMels")

    # ===== Vocoder =====
    local_dir = snapshot_download(
        repo_id="amphion/Vevo",
        repo_type="model",
        cache_dir="./ckpts/Vevo",
        allow_patterns=["acoustic_modeling/Vocoder/*"],
    )

    vocoder_cfg_path = "./models/vc/vevo/config/Vocoder.json"
    vocoder_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vocoder")

    # ===== Inference =====
    inference_pipeline = VevoInferencePipeline(
        content_style_tokenizer_ckpt_path=content_style_tokenizer_ckpt_path,
        ar_cfg_path=ar_cfg_path,
        ar_ckpt_path=ar_ckpt_path,
        fmt_cfg_path=fmt_cfg_path,
        fmt_ckpt_path=fmt_ckpt_path,
        vocoder_cfg_path=vocoder_cfg_path,
        vocoder_ckpt_path=vocoder_ckpt_path,
        device=device,
    )

    src_text = "I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences."

    ref_wav_path = "./models/vc/vevo/wav/arabic_male.wav"
    ref_text = "Flip stood undecided, his ears strained to catch the slightest sound."

    # 1. Zero-Shot TTS (the style reference and timbre reference are same)
    vevo_tts(
        src_text,
        ref_wav_path,
        output_path="./models/vc/vevo/wav/output_vevotts1.wav",
        ref_text=ref_text,
        src_language="en",
        ref_language="en",
    )

    # 2. Style and Timbre Controllable Zero-Shot TTS (the style reference and timbre reference are different)
    vevo_tts(
        src_text,
        ref_wav_path,
        timbre_ref_wav_path="./models/vc/vevo/wav/mandarin_female.wav",
        output_path="./models/vc/vevo/wav/output_vevotts2.wav",
        ref_text=ref_text,
        src_language="en",
        ref_language="en",
    )
```

## Citation

If you use Vevo in your research, please cite the following papers:

```bibtex
@inproceedings{vevo,
  author       = {Xueyao Zhang and Xiaohui Zhang and Kainan Peng and Zhenyu Tang and Vimal Manohar and Yingru Liu and Jeff Hwang and Dangna Li and Yuhao Wang and Julian Chan and Yuan Huang and Zhizheng Wu and Mingbo Ma},
  title        = {Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
  booktitle    = {{ICLR}},
  publisher    = {OpenReview.net},
  year         = {2025}
}

@article{amphion_v0.2,
  title        = {Overview of the Amphion Toolkit (v0.2)},
  author       = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu},
  year         = {2025},
  journal      = {arXiv preprint arXiv:2501.15442},
}

@inproceedings{amphion,
    author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
    title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
    booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
    year={2024}
}
```