|
Download README.md from moonbox-dev/Vevo-ONNX: direct link, hf CLI and curl.
- Browser
- Download file 17.8 kB
-
https://huggingface.co/moonbox-dev/Vevo-ONNX/resolve/main/README.md
- Command line
-
hf download hf://moonbox-dev/Vevo-ONNX/README.md
-
curl -L -o README.md https://huggingface.co/moonbox-dev/Vevo-ONNX/resolve/main/README.md
17.8 kB
| base_model: amphion/Vevo | |
| base_model_relation: quantized | |
| license: cc-by-nc-4.0 | |
| pipeline_tag: audio-to-audio | |
| library_name: onnx | |
| tags: | |
| - onnx | |
| - moonbox | |
| - onnxruntime | |
| - vevo | |
| - amphion | |
| - voice-conversion | |
| - zero-shot | |
| - non-commercial | |
| # Vevo ONNX | |
| ONNX builds of [Vevo](https://huggingface.co/amphion/Vevo), made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each | |
| precision is one folder holding the graphs and a small Python `pipeline.py` that does the rest (features, | |
| resampling, the conversion steps); what the graphs take and return is under *How to run*. | |
| ## Files | |
| The last column is the benchmark on Moonbox's voice-conversion set (English LibriSpeech and Chinese AISHELL-1 | |
| recordings, each converted into an English and a Chinese target voice; details under *How to run*): error rate of a | |
| Whisper transcript of the output against the source's transcript, speaker similarity to the target voice and DNSMOS, | |
| against the original PyTorch model. | |
| | Precision | File | Size | sha256 | Benchmark | | |
| |---|---|---|---|---| | |
| | int8 | [`int8/cond.onnx`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/cond.onnx) | 33.6 MB | `25c2af7d480f` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) | | |
| | int8 | [`int8/config.json`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/config.json) | 0.0 kB | `21abd1bc2f85` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) | | |
| | int8 | [`int8/content.onnx`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/content.onnx) | 0.3 MB | `2496a23dcfbc` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) | | |
| | int8 | [`int8/content.onnx.data`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/content.onnx.data) | 1.05 GB | `68c51fe1ac74` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) | | |
| | int8 | [`int8/estimator.onnx`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/estimator.onnx) | 331.2 MB | `aab0060de5a5` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) | | |
| | int8 | [`int8/mel_basis.npy`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/mel_basis.npy) | 0.5 MB | `7bb7e7de4592` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) | | |
| | int8 | [`int8/pipeline.py`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/pipeline.py) | 7.2 kB | `b87a739f78be` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) | | |
| | int8 | [`int8/vocoder.onnx`](https://huggingface.co/moonbox-dev/Vevo-ONNX/blob/main/int8/vocoder.onnx) | 257.0 MB | `995abcdb8559` | content fp32, rest int8: WER 1.1 % / CER 4.2 % (original 1.1 / 4.2, fp32), similarity 0.908 / 0.899 (0.907 / 0.893), DNSMOS 2.91 / 2.90 (2.92 / 2.93); SNR 8 dB vs original; RTF 0.44 (CUDA) | | |
| ## How to run | |
| Vevo-Timbre (Amphion's Vevo: zero-shot voice conversion that re-voices a recording into the timbre of a few seconds of | |
| reference audio, keeping its words, timing and speaking style; 24 kHz output) as four ONNX graphs plus `pipeline.py`, | |
| a numpy runtime that reproduces the official `VevoInferencePipeline.inference_fm` (`infer_vevotimbre.py`: 32 | |
| flow-matching steps, classifier-free guidance 1.0 rescaled 0.75) around them. Vevo's autoregressive models | |
| (Vevo-Voice and Vevo-Style, which also convert accent and emotion, and Vevo-TTS) are not included. | |
| - `content.onnx`: 16 kHz audio `wave` [1, N] -> content-style tokens `tokens` [1, N / 320] (50 Hz, 8192 codes: | |
| torchaudio's HuBERT-large layers 1-18, mean / std normalisation, the RepCodec encoder and vector quantizer). | |
| - `cond.onnx`: tokens [1, L] -> the flow's condition `cond` [1, L, 1024]. | |
| - `estimator.onnx`: the flow-matching transformer: `x` [B, L, 128] (reference mel, then the noisy target), `t` [B], | |
| `cond` [B, L, 1024] -> `flow` [B, L, 128]. | |
| - `vocoder.onnx`: the Vocos vocoder up to its ISTFT head: normalised mel `mel` [1, 128, T] -> STFT `real`, `imag` | |
| [1, 961, T]. `pipeline.py` does the inverse STFT (n_fft 1920, hop 480, "same" padding) in numpy. | |
| `pipeline.py` also does the resampling (soxr to 24 kHz, then torchaudio's sinc resampler ported to numpy for the 16 | |
| kHz tokenizer input, as the official `load_wav` does) and the 128-bin log-mel of the reference (`mel_basis.npy`, | |
| normalised with `config.json`). The flow's starting noise comes from numpy's generator, so a seed gives the same | |
| output every run here (not the same as PyTorch with that seed). The flow attends over reference and source together, | |
| with no chunking: keep the reference to a few seconds and the source under about 30 s, as the official examples do. | |
| It needs numpy, onnxruntime (or onnxruntime-gpu), soundfile and soxr: | |
| ```bash | |
| python int8/pipeline.py source.wav voice.wav out.wav --cuda --seed 0 | |
| ``` | |
| ```python | |
| import sys, soundfile as sf | |
| sys.path.insert(0, "int8") | |
| import pipeline | |
| vc = pipeline.VevoTimbre("int8", "cuda") | |
| src, sr = sf.read("source.wav", dtype="float32") | |
| ref, ref_sr = sf.read("voice.wav", dtype="float32") | |
| wav, out_sr = vc(src, sr, ref, ref_sr, steps=32, seed=0) # 24 kHz | |
| ``` | |
| **Precisions.** `int8/` only (1.6 GB): weight-only int8 for the flow transformer, the vocoder and the condition | |
| embedding, the content tokenizer in fp32 (1.26 GB of it). An fp16 tokenizer flips enough of its 8192 codes to put | |
| Chinese CER at 22.9 % against the original's 4.2 %. With the tokenizer in fp32, fp16 math in the flow and vocoder still | |
| gave 8.3 %, and a 4-bit flow transformer gave 14.6 %, so there is no `fp16/` or `int4/`. No bf16 build. | |
| **Benchmark.** The lab's voice-conversion set: 4 English recordings (LibriSpeech test-clean) and 2 Chinese ones | |
| (AISHELL-1 test), each converted into the English (LibriSpeech) and the Chinese (AISHELL-1) voice of the lab's TTS | |
| prompts: 12 conversions. Each output is judged by Whisper large-v3-turbo's WER (English sources) or CER (Chinese) | |
| against the source's transcript, WavLM-base-plus-sv speaker similarity to the target voice, and DNSMOS (P.835 | |
| overall). The original is the official code (Amphion @ 26f6883, `inference_fm`) in fp32 on CUDA, given the same | |
| starting noise. Pass, per source language: error rate at most 2 points above the original's, similarity at most | |
| 0.03 below, DNSMOS at most 0.2 below. Real-time factor (processing time / audio time) on an RTX 3080 Laptop GPU, | |
| whole pipeline. SNR is each output against the original's on the same input and noise. | |
| | | English WER | similarity | DNSMOS | Chinese CER | similarity | DNSMOS | SNR vs original | Real-time factor | | |
| |---|---|---|---|---|---|---|---|---| | |
| | original (PyTorch fp32, CUDA) | 1.1 % | 0.907 | 2.92 | 4.2 % | 0.893 | 2.93 | | 0.54 | | |
| | `int8/` (CUDA) | 1.1 % | 0.908 | 2.91 | 4.2 % | 0.899 | 2.90 | 8 dB | 0.44 | | |
| The Chinese set is small (4 clips, about 50 characters: one character is 2 points). The graphs match the PyTorch | |
| modules on test inputs (content tokens identical, the flow transformer 4e-6, vocoder with the numpy inverse STFT 9e-8); | |
| the fp32 graphs reach 11 dB SNR against the original on this set, where 32 guided flow steps carry small differences | |
| into the waveform. | |
| **License.** The Vevo weights are CC-BY-NC-4.0 (amphion/Vevo): these files are too, so they may not be used | |
| commercially; credit Amphion's Vevo (Zhang et al., "Vevo: Controllable Zero-Shot Voice Imitation with | |
| Self-Supervised Disentanglement", ICLR 2025). The Amphion code they were exported from is MIT. The tokenizer contains | |
| HuBERT-large from torchaudio's HUBERT_LARGE bundle (fairseq, MIT). | |
| ## How these were made | |
| From [Vevo](https://huggingface.co/amphion/Vevo) at revision `7edf4640c4`. The graphs were exported from the original PyTorch modules with | |
| torch.onnx (opset 17 / 18) and checked against them. Weight-only quantization: int8 is a per-output-channel int8 | |
| weight behind DequantizeLinear, int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was | |
| benchmarked against the original before upload (pass, per language: error rate at most 2 points above the | |
| original's, similarity to the target voice at most 0.03 below, DNSMOS at most 0.2 below), and the `sha256` column above is what | |
| the files must hash to. | |
| ## License | |
| CC-BY-NC-4.0. These files are a derivative of the original model and carry its license: it is copied in this | |
| repository, and it governs your use of these files. The original model card follows unchanged. | |
| --- | |
| # Original model card | |
| # Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement | |
| [](https://openreview.net/pdf?id=anQDiQZhDP) | |
| [](https://huggingface.co/amphion/Vevo) | |
| [](https://versavoice.github.io/) [](https://github.com/open-mmlab/Amphion/blob/main/models/vc/vevo/README.md) | |
| We present our reproduction of [Vevo](https://openreview.net/pdf?id=anQDiQZhDP), a versatile zero-shot voice imitation framework with controllable timbre and style. We invite you to explore the [audio samples](https://versavoice.github.io/) to experience Vevo's capabilities firsthand. | |
| We have included the following pre-trained Vevo models at Amphion: | |
| - **Vevo-Timbre**: It can conduct *style-preserved* voice conversion. | |
| - **Vevo-Style**: It can conduct style conversion, such as *accent conversion* and *emotion conversion*. | |
| - **Vevo-Voice**: It can conduct *style-converted* voice conversion. | |
| - **Vevo-TTS**: It can conduct *style and timbre controllable* TTS. | |
| Besides, we also release the **content tokenizer** and **content-style tokenizer** proposed by Vevo. Notably, all these pre-trained models are trained on [Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset), containing 101k hours of speech data among six languages (English, Chinese, German, French, Japanese, and Korean). | |
| ## Model Introduction | |
| We provide the following pre-trained models: | |
| | Model Name | Description | | |
| |-------------------|-------------| | |
| | [Content Tokenizer](https://huggingface.co/amphion/Vevo/tree/main/tokenizer/vq32) | Converting speech to content tokens. It is a single codebook VQ-VAE with a vocabulary size of 32. The frame rate is 50Hz.| | |
| | [Content-Style Tokenizer](https://huggingface.co/amphion/Vevo/tree/main/tokenizer/vq8192) | Converting speech to content-style tokens. It is a single codebook VQ-VAE with a vocabulary size of 8192. The frame rate is 50Hz.| | |
| | [Vq32ToVq8192](https://huggingface.co/amphion/Vevo/tree/main/contentstyle_modeling/Vq32ToVq8192) | Predicting content-style tokens from content tokens with an auto-regressive transformer (480M). | | |
| | [PhoneToVq8192](https://huggingface.co/amphion/Vevo/tree/main/contentstyle_modeling/PhoneToVq8192) | Predicting content-style tokens from phone tokens with an auto-regressive transformer (740M). | | |
| | [Vq8192ToMels](https://huggingface.co/amphion/Vevo/tree/main/acoustic_modeling/Vq8192ToMels) | Predicting mel-spectrogram from content-style tokens with a flow-matching transformer (330M). | | |
| | [Vocoder](https://huggingface.co/amphion/Vevo/tree/main/acoustic_modeling/Vocoder) | Predicting audio from mel-spectrogram with a Vocos-based vocoder (250M). | | |
| You can download all pretrained checkpoints from [HuggingFace](https://huggingface.co/amphion/MaskGCT/tree/main) or use huggingface API. | |
| ## Usage | |
| You can refer to our [recipe](https://github.com/open-mmlab/Amphion/blob/main/models/vc/vevo/README.md) at GitHub for more usage details. For example, to use Vevo-TTS, after you clone the Amphion github repository, you can use the script like: | |
| ```python | |
| import os | |
| from huggingface_hub import snapshot_download | |
| from models.vc.vevo.vevo_utils import * | |
| def vevo_tts( | |
| src_text, | |
| ref_wav_path, | |
| timbre_ref_wav_path=None, | |
| output_path=None, | |
| ref_text=None, | |
| src_language="en", | |
| ref_language="en", | |
| ): | |
| if timbre_ref_wav_path is None: | |
| timbre_ref_wav_path = ref_wav_path | |
| gen_audio = inference_pipeline.inference_ar_and_fm( | |
| src_wav_path=None, | |
| src_text=src_text, | |
| style_ref_wav_path=ref_wav_path, | |
| timbre_ref_wav_path=timbre_ref_wav_path, | |
| style_ref_wav_text=ref_text, | |
| src_text_language=src_language, | |
| style_ref_wav_text_language=ref_language, | |
| ) | |
| assert output_path is not None | |
| save_audio(gen_audio, output_path=output_path) | |
| if __name__ == "__main__": | |
| # ===== Device ===== | |
| device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu") | |
| # ===== Content-Style Tokenizer ===== | |
| local_dir = snapshot_download( | |
| repo_id="amphion/Vevo", | |
| repo_type="model", | |
| cache_dir="./ckpts/Vevo", | |
| allow_patterns=["tokenizer/vq8192/*"], | |
| ) | |
| content_style_tokenizer_ckpt_path = os.path.join(local_dir, "tokenizer/vq8192") | |
| # ===== Autoregressive Transformer ===== | |
| local_dir = snapshot_download( | |
| repo_id="amphion/Vevo", | |
| repo_type="model", | |
| cache_dir="./ckpts/Vevo", | |
| allow_patterns=["contentstyle_modeling/PhoneToVq8192/*"], | |
| ) | |
| ar_cfg_path = "./models/vc/vevo/config/PhoneToVq8192.json" | |
| ar_ckpt_path = os.path.join(local_dir, "contentstyle_modeling/PhoneToVq8192") | |
| # ===== Flow Matching Transformer ===== | |
| local_dir = snapshot_download( | |
| repo_id="amphion/Vevo", | |
| repo_type="model", | |
| cache_dir="./ckpts/Vevo", | |
| allow_patterns=["acoustic_modeling/Vq8192ToMels/*"], | |
| ) | |
| fmt_cfg_path = "./models/vc/vevo/config/Vq8192ToMels.json" | |
| fmt_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vq8192ToMels") | |
| # ===== Vocoder ===== | |
| local_dir = snapshot_download( | |
| repo_id="amphion/Vevo", | |
| repo_type="model", | |
| cache_dir="./ckpts/Vevo", | |
| allow_patterns=["acoustic_modeling/Vocoder/*"], | |
| ) | |
| vocoder_cfg_path = "./models/vc/vevo/config/Vocoder.json" | |
| vocoder_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vocoder") | |
| # ===== Inference ===== | |
| inference_pipeline = VevoInferencePipeline( | |
| content_style_tokenizer_ckpt_path=content_style_tokenizer_ckpt_path, | |
| ar_cfg_path=ar_cfg_path, | |
| ar_ckpt_path=ar_ckpt_path, | |
| fmt_cfg_path=fmt_cfg_path, | |
| fmt_ckpt_path=fmt_ckpt_path, | |
| vocoder_cfg_path=vocoder_cfg_path, | |
| vocoder_ckpt_path=vocoder_ckpt_path, | |
| device=device, | |
| ) | |
| src_text = "I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences." | |
| ref_wav_path = "./models/vc/vevo/wav/arabic_male.wav" | |
| ref_text = "Flip stood undecided, his ears strained to catch the slightest sound." | |
| # 1. Zero-Shot TTS (the style reference and timbre reference are same) | |
| vevo_tts( | |
| src_text, | |
| ref_wav_path, | |
| output_path="./models/vc/vevo/wav/output_vevotts1.wav", | |
| ref_text=ref_text, | |
| src_language="en", | |
| ref_language="en", | |
| ) | |
| # 2. Style and Timbre Controllable Zero-Shot TTS (the style reference and timbre reference are different) | |
| vevo_tts( | |
| src_text, | |
| ref_wav_path, | |
| timbre_ref_wav_path="./models/vc/vevo/wav/mandarin_female.wav", | |
| output_path="./models/vc/vevo/wav/output_vevotts2.wav", | |
| ref_text=ref_text, | |
| src_language="en", | |
| ref_language="en", | |
| ) | |
| ``` | |
| ## Citation | |
| If you use Vevo in your research, please cite the following papers: | |
| ```bibtex | |
| @inproceedings{vevo, | |
| author = {Xueyao Zhang and Xiaohui Zhang and Kainan Peng and Zhenyu Tang and Vimal Manohar and Yingru Liu and Jeff Hwang and Dangna Li and Yuhao Wang and Julian Chan and Yuan Huang and Zhizheng Wu and Mingbo Ma}, | |
| title = {Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement}, | |
| booktitle = {{ICLR}}, | |
| publisher = {OpenReview.net}, | |
| year = {2025} | |
| } | |
| @article{amphion_v0.2, | |
| title = {Overview of the Amphion Toolkit (v0.2)}, | |
| author = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu}, | |
| year = {2025}, | |
| journal = {arXiv preprint arXiv:2501.15442}, | |
| } | |
| @inproceedings{amphion, | |
| author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu}, | |
| title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit}, | |
| booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024}, | |
| year={2024} | |
| } | |
| ``` | |