Vevo ONNX

ONNX builds of Vevo, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder holding the graphs and a small Python pipeline.py that does the rest (features, resampling, the conversion steps); what the graphs take and return is under How to run.

Files

The last column is the benchmark on Moonbox's voice-conversion set (English LibriSpeech and Chinese AISHELL-1 recordings, each converted into an English and a Chinese target voice; details under How to run): error rate of a Whisper transcript of the output against the source's transcript, speaker similarity to the target voice and DNSMOS, against the original PyTorch model.

Precision File Size sha256 Benchmark
int8 int8/cond.onnx 33.6 MB 25c2af7d480f content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA)
int8 int8/config.json 0.0 kB 21abd1bc2f85 content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA)
int8 int8/content.onnx 0.3 MB 2496a23dcfbc content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA)
int8 int8/content.onnx.data 1.05 GB 68c51fe1ac74 content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA)
int8 int8/estimator.onnx 331.2 MB aab0060de5a5 content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA)
int8 int8/mel_basis.npy 0.5 MB 7bb7e7de4592 content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA)
int8 int8/pipeline.py 7.2 kB b87a739f78be content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA)
int8 int8/vocoder.onnx 257.0 MB 995abcdb8559 content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA)

How to run

Vevo-Timbre (Amphion's Vevo: zero-shot voice conversion that re-voices a recording into the timbre of a few seconds of reference audio, keeping its words, timing and speaking style; 24 kHz output) as four ONNX graphs plus pipeline.py, a numpy runtime that reproduces the official VevoInferencePipeline.inference_fm (infer_vevotimbre.py: 32 flow-matching steps, classifier-free guidance 1.0 rescaled 0.75) around them. Vevo's autoregressive models (Vevo-Voice and Vevo-Style, which also convert accent and emotion, and Vevo-TTS) are not included.

  • content.onnx: 16 kHz audio wave [1, N] -> content-style tokens tokens [1, N / 320] (50 Hz, 8192 codes: torchaudio's HuBERT-large layers 1-18, mean / std normalisation, the RepCodec encoder and vector quantizer).
  • cond.onnx: tokens [1, L] -> the flow's condition cond [1, L, 1024].
  • estimator.onnx: the flow-matching transformer: x [B, L, 128] (reference mel, then the noisy target), t [B], cond [B, L, 1024] -> flow [B, L, 128].
  • vocoder.onnx: the Vocos vocoder up to its ISTFT head: normalised mel mel [1, 128, T] -> STFT real, imag [1, 961, T]. pipeline.py does the inverse STFT (n_fft 1920, hop 480, "same" padding) in numpy.

pipeline.py also does the resampling (soxr to 24 kHz, then torchaudio's sinc resampler ported to numpy for the 16 kHz tokenizer input, as the official load_wav does) and the 128-bin log-mel of the reference (mel_basis.npy, normalised with config.json). The flow's starting noise comes from numpy's generator, so a seed gives the same output every run here (not the same as PyTorch with that seed). The flow attends over reference and source together, with no chunking: keep the reference to a few seconds and the source under about 30 s, as the official examples do. It needs numpy, onnxruntime (or onnxruntime-gpu), soundfile and soxr:

python int8/pipeline.py source.wav voice.wav out.wav --cuda --seed 0
import sys, soundfile as sf
sys.path.insert(0, "int8")
import pipeline
vc = pipeline.VevoTimbre("int8", "cuda")
src, sr = sf.read("source.wav", dtype="float32")
ref, ref_sr = sf.read("voice.wav", dtype="float32")
wav, out_sr = vc(src, sr, ref, ref_sr, steps=32, seed=0)       # 24 kHz

Precisions. int8/ only (1.6 GB): weight-only int8 for the flow transformer, the vocoder and the condition embedding, the content tokenizer in fp32 (1.26 GB of it). An fp16 tokenizer flips enough of its 8192 codes to put Chinese CER at 22.9 % against the original's 4.2 %. With the tokenizer in fp32, fp16 math in the flow and vocoder still gave 8.3 %, and a 4-bit flow transformer gave 14.6 %, so there is no fp16/ or int4/. No bf16 build.

Benchmark. The lab's voice-conversion set: 4 English recordings (LibriSpeech test-clean) and 2 Chinese ones (AISHELL-1 test), each converted into the English (LibriSpeech) and the Chinese (AISHELL-1) voice of the lab's TTS prompts: 12 conversions. Each output is judged by Whisper large-v3-turbo's WER (English sources) or CER (Chinese) against the source's transcript, WavLM-base-plus-sv speaker similarity to the target voice, and DNSMOS (P.835 overall). The original is the official code (Amphion @ 26f6883, inference_fm) in fp32 on CUDA, given the same starting noise. Pass, per source language: error rate at most 2 points above the original's, similarity at most 0.03 below, DNSMOS at most 0.2 below. Real-time factor (processing time / audio time) on an RTX 3080 Laptop GPU, whole pipeline. SNR is each output against the original's on the same input and noise.

English WER similarity DNSMOS Chinese CER similarity DNSMOS SNR vs original Real-time factor
original (PyTorch fp32, CUDA) 1.1 % 0.907 2.92 4.2 % 0.893 2.93 0.54
int8/ (CUDA) 1.1 % 0.908 2.91 4.2 % 0.899 2.90 8 dB 0.44

The Chinese set is small (4 clips, about 50 characters: one character is 2 points). The graphs match the PyTorch modules on test inputs (content tokens identical, the flow transformer 4e-6, vocoder with the numpy inverse STFT 9e-8); the fp32 graphs reach 11 dB SNR against the original on this set, where 32 guided flow steps carry small differences into the waveform.

License. The Vevo weights are CC-BY-NC-4.0 (amphion/Vevo): these files are too, so they may not be used commercially; credit Amphion's Vevo (Zhang et al., "Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement", ICLR 2025). The Amphion code they were exported from is MIT. The tokenizer contains HuBERT-large from torchaudio's HUBERT_LARGE bundle (fairseq, MIT).

How these were made

From Vevo at revision 7edf4640c4. The graphs were exported from the original PyTorch modules with torch.onnx (opset 17 / 18) and checked against them. Weight-only quantization: int8 is a per-output-channel int8 weight behind DequantizeLinear, int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was benchmarked against the original before upload (pass, per language: error rate at most 2 points above the original's, similarity to the target voice at most 0.03 below, DNSMOS at most 0.2 below), and the sha256 column above is what the files must hash to.

License

CC-BY-NC-4.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.


Original model card

Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

arXiv hf WebPage readme

We present our reproduction of Vevo, a versatile zero-shot voice imitation framework with controllable timbre and style. We invite you to explore the audio samples to experience Vevo's capabilities firsthand.

We have included the following pre-trained Vevo models at Amphion:

  • Vevo-Timbre: It can conduct style-preserved voice conversion.
  • Vevo-Style: It can conduct style conversion, such as accent conversion and emotion conversion.
  • Vevo-Voice: It can conduct style-converted voice conversion.
  • Vevo-TTS: It can conduct style and timbre controllable TTS.

Besides, we also release the content tokenizer and content-style tokenizer proposed by Vevo. Notably, all these pre-trained models are trained on Emilia, containing 101k hours of speech data among six languages (English, Chinese, German, French, Japanese, and Korean).

Model Introduction

We provide the following pre-trained models:

Model Name Description
Content Tokenizer Converting speech to content tokens. It is a single codebook VQ-VAE with a vocabulary size of 32. The frame rate is 50Hz.
Content-Style Tokenizer Converting speech to content-style tokens. It is a single codebook VQ-VAE with a vocabulary size of 8192. The frame rate is 50Hz.
Vq32ToVq8192 Predicting content-style tokens from content tokens with an auto-regressive transformer (480M).
PhoneToVq8192 Predicting content-style tokens from phone tokens with an auto-regressive transformer (740M).
Vq8192ToMels Predicting mel-spectrogram from content-style tokens with a flow-matching transformer (330M).
Vocoder Predicting audio from mel-spectrogram with a Vocos-based vocoder (250M).

You can download all pretrained checkpoints from HuggingFace or use huggingface API.

Usage

You can refer to our recipe at GitHub for more usage details. For example, to use Vevo-TTS, after you clone the Amphion github repository, you can use the script like:

import os
from huggingface_hub import snapshot_download

from models.vc.vevo.vevo_utils import *


def vevo_tts(
    src_text,
    ref_wav_path,
    timbre_ref_wav_path=None,
    output_path=None,
    ref_text=None,
    src_language="en",
    ref_language="en",
):
    if timbre_ref_wav_path is None:
        timbre_ref_wav_path = ref_wav_path

    gen_audio = inference_pipeline.inference_ar_and_fm(
        src_wav_path=None,
        src_text=src_text,
        style_ref_wav_path=ref_wav_path,
        timbre_ref_wav_path=timbre_ref_wav_path,
        style_ref_wav_text=ref_text,
        src_text_language=src_language,
        style_ref_wav_text_language=ref_language,
    )

    assert output_path is not None
    save_audio(gen_audio, output_path=output_path)


if __name__ == "__main__":
    # ===== Device =====
    device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

    # ===== Content-Style Tokenizer =====
    local_dir = snapshot_download(
        repo_id="amphion/Vevo",
        repo_type="model",
        cache_dir="./ckpts/Vevo",
        allow_patterns=["tokenizer/vq8192/*"],
    )

    content_style_tokenizer_ckpt_path = os.path.join(local_dir, "tokenizer/vq8192")

    # ===== Autoregressive Transformer =====
    local_dir = snapshot_download(
        repo_id="amphion/Vevo",
        repo_type="model",
        cache_dir="./ckpts/Vevo",
        allow_patterns=["contentstyle_modeling/PhoneToVq8192/*"],
    )

    ar_cfg_path = "./models/vc/vevo/config/PhoneToVq8192.json"
    ar_ckpt_path = os.path.join(local_dir, "contentstyle_modeling/PhoneToVq8192")

    # ===== Flow Matching Transformer =====
    local_dir = snapshot_download(
        repo_id="amphion/Vevo",
        repo_type="model",
        cache_dir="./ckpts/Vevo",
        allow_patterns=["acoustic_modeling/Vq8192ToMels/*"],
    )

    fmt_cfg_path = "./models/vc/vevo/config/Vq8192ToMels.json"
    fmt_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vq8192ToMels")

    # ===== Vocoder =====
    local_dir = snapshot_download(
        repo_id="amphion/Vevo",
        repo_type="model",
        cache_dir="./ckpts/Vevo",
        allow_patterns=["acoustic_modeling/Vocoder/*"],
    )

    vocoder_cfg_path = "./models/vc/vevo/config/Vocoder.json"
    vocoder_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vocoder")

    # ===== Inference =====
    inference_pipeline = VevoInferencePipeline(
        content_style_tokenizer_ckpt_path=content_style_tokenizer_ckpt_path,
        ar_cfg_path=ar_cfg_path,
        ar_ckpt_path=ar_ckpt_path,
        fmt_cfg_path=fmt_cfg_path,
        fmt_ckpt_path=fmt_ckpt_path,
        vocoder_cfg_path=vocoder_cfg_path,
        vocoder_ckpt_path=vocoder_ckpt_path,
        device=device,
    )

    src_text = "I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences."

    ref_wav_path = "./models/vc/vevo/wav/arabic_male.wav"
    ref_text = "Flip stood undecided, his ears strained to catch the slightest sound."

    # 1. Zero-Shot TTS (the style reference and timbre reference are same)
    vevo_tts(
        src_text,
        ref_wav_path,
        output_path="./models/vc/vevo/wav/output_vevotts1.wav",
        ref_text=ref_text,
        src_language="en",
        ref_language="en",
    )

    # 2. Style and Timbre Controllable Zero-Shot TTS (the style reference and timbre reference are different)
    vevo_tts(
        src_text,
        ref_wav_path,
        timbre_ref_wav_path="./models/vc/vevo/wav/mandarin_female.wav",
        output_path="./models/vc/vevo/wav/output_vevotts2.wav",
        ref_text=ref_text,
        src_language="en",
        ref_language="en",
    )

Citation

If you use Vevo in your research, please cite the following papers:

@inproceedings{vevo,
  author       = {Xueyao Zhang and Xiaohui Zhang and Kainan Peng and Zhenyu Tang and Vimal Manohar and Yingru Liu and Jeff Hwang and Dangna Li and Yuhao Wang and Julian Chan and Yuan Huang and Zhizheng Wu and Mingbo Ma},
  title        = {Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
  booktitle    = {{ICLR}},
  publisher    = {OpenReview.net},
  year         = {2025}
}

@article{amphion_v0.2,
  title        = {Overview of the Amphion Toolkit (v0.2)},
  author       = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu},
  year         = {2025},
  journal      = {arXiv preprint arXiv:2501.15442},
}

@inproceedings{amphion,
    author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
    title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
    booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
    year={2024}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for moonbox-dev/Vevo-ONNX

Base model

amphion/Vevo
Quantized
(1)
this model

Paper for moonbox-dev/Vevo-ONNX