|
Download README.md from moonbox-dev/OpenVoiceV2-ONNX: direct link, hf CLI and curl.
- Browser
- Download file 13.3 kB
-
https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/resolve/main/README.md
- Command line
-
hf download hf://moonbox-dev/OpenVoiceV2-ONNX/README.md
-
curl -L -o README.md https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/resolve/main/README.md
13.3 kB
| base_model: myshell-ai/OpenVoiceV2 | |
| base_model_relation: quantized | |
| license: mit | |
| pipeline_tag: audio-to-audio | |
| library_name: onnx | |
| tags: | |
| - onnx | |
| - moonbox | |
| - onnxruntime | |
| - openvoice | |
| - voice-conversion | |
| - tone-color-converter | |
| - multilingual | |
| # OpenVoice V2 ONNX | |
| ONNX builds of [OpenVoice V2](https://huggingface.co/myshell-ai/OpenVoiceV2), made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each | |
| precision is one folder holding the graphs and a small Python `pipeline.py` that does the rest (features, | |
| resampling, the conversion steps); what the graphs take and return is under *How to run*. | |
| ## Files | |
| The last column is the benchmark on Moonbox's voice-conversion set (English LibriSpeech and Chinese AISHELL-1 | |
| recordings, each converted into an English and a Chinese target voice; details under *How to run*): error rate of a | |
| Whisper transcript of the output against the source's transcript, speaker similarity to the target voice and DNSMOS, | |
| against the original PyTorch model. | |
| | Precision | File | Size | sha256 | Benchmark | | |
| |---|---|---|---|---| | |
| | fp16 | [`fp16/config.json`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/fp16/config.json) | 0.8 kB | `9dfff60350b8` | WER 2.6 % / CER 25.0 % (original 2.6 / 27.1, fp32), similarity 0.842 / 0.792 (0.842 / 0.797), DNSMOS 3.31 / 2.90 (3.30 / 2.89); SNR 23 dB vs original; RTF 0.06 (CUDA) | | |
| | fp16 | [`fp16/converter.onnx`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/fp16/converter.onnx) | 64.3 MB | `22a79ab872d1` | WER 2.6 % / CER 25.0 % (original 2.6 / 27.1, fp32), similarity 0.842 / 0.792 (0.842 / 0.797), DNSMOS 3.31 / 2.90 (3.30 / 2.89); SNR 23 dB vs original; RTF 0.06 (CUDA) | | |
| | fp16 | [`fp16/pipeline.py`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/fp16/pipeline.py) | 3.7 kB | `876563834920` | WER 2.6 % / CER 25.0 % (original 2.6 / 27.1, fp32), similarity 0.842 / 0.792 (0.842 / 0.797), DNSMOS 3.31 / 2.90 (3.30 / 2.89); SNR 23 dB vs original; RTF 0.06 (CUDA) | | |
| | fp16 | [`fp16/ref_encoder.onnx`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/fp16/ref_encoder.onnx) | 1.6 MB | `50742432520f` | WER 2.6 % / CER 25.0 % (original 2.6 / 27.1, fp32), similarity 0.842 / 0.792 (0.842 / 0.797), DNSMOS 3.31 / 2.90 (3.30 / 2.89); SNR 23 dB vs original; RTF 0.06 (CUDA) | | |
| | int8 | [`int8/config.json`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/int8/config.json) | 0.8 kB | `9dfff60350b8` | WER 2.6 % / CER 25.0 %, similarity 0.845 / 0.789, DNSMOS 3.30 / 2.89; SNR 5 dB vs original; RTF 0.04 (CUDA) | | |
| | int8 | [`int8/converter.onnx`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/int8/converter.onnx) | 32.7 MB | `17c254cb33ca` | WER 2.6 % / CER 25.0 %, similarity 0.845 / 0.789, DNSMOS 3.30 / 2.89; SNR 5 dB vs original; RTF 0.04 (CUDA) | | |
| | int8 | [`int8/pipeline.py`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/int8/pipeline.py) | 3.7 kB | `876563834920` | WER 2.6 % / CER 25.0 %, similarity 0.845 / 0.789, DNSMOS 3.30 / 2.89; SNR 5 dB vs original; RTF 0.04 (CUDA) | | |
| | int8 | [`int8/ref_encoder.onnx`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/int8/ref_encoder.onnx) | 0.8 MB | `2e8b8b34c880` | WER 2.6 % / CER 25.0 %, similarity 0.845 / 0.789, DNSMOS 3.30 / 2.89; SNR 5 dB vs original; RTF 0.04 (CUDA) | | |
| ## How to run | |
| OpenVoice V2's tone-colour converter (MyShell: re-voices a recording into the voice of a reference recording, | |
| keeping its words and timing; any language, 22.05 kHz output) as two ONNX graphs plus `pipeline.py`, a numpy runtime | |
| that reproduces the official `ToneColorConverter.extract_se` / `convert(tau=0.3)` around them. The base speakers | |
| (MeloTTS) are not part of this repository: give it any recording as the source. | |
| - `ref_encoder.onnx`: linear spectrogram `spec` [1, T, 513] (22.05 kHz, n_fft 1024, hop 256) -> tone-colour | |
| embedding `se` [1, 256]. | |
| - `converter.onnx`: the source's spectrogram `spec` [1, 513, T], the posterior encoder's sampling noise `noise` | |
| [1, 192, T], and the source and target embeddings `src_se`, `tgt_se` [1, 256, 1] -> audio `wav` [1, 1, 256 T] | |
| (the original `SynthesizerTrn.voice_conversion` with tau 0.3). | |
| The noise is an input, so a seed gives the same output every run. `pipeline.py` computes the spectrogram in numpy | |
| (the original's `spectrogram_torch(center=False)`), resamples with soxr, and embeds the source and the target voice | |
| each in one piece (the official `se_extractor.get_se` first splits the reference with a VAD and averages; for a long | |
| reference you can do the same with `embed` and average the embeddings). The watermark the official code can add is | |
| not included. It needs numpy, onnxruntime (or onnxruntime-gpu), soundfile and soxr: | |
| ```bash | |
| python fp16/pipeline.py source.wav voice.wav out.wav --cuda --seed 0 | |
| ``` | |
| ```python | |
| import sys, soundfile as sf | |
| sys.path.insert(0, "fp16") | |
| import pipeline | |
| vc = pipeline.OpenVoiceV2("fp16", "cuda") | |
| src, sr = sf.read("source.wav", dtype="float32") | |
| ref, ref_sr = sf.read("voice.wav", dtype="float32") | |
| wav, out_sr = vc(src, sr, ref, ref_sr, seed=0) # 22.05 kHz | |
| ``` | |
| **Precisions.** `fp16/`: both graphs fp16 math with fp32 inputs and outputs (66 MB). `int8/`: weight-only int8 for | |
| every convolution, GRU and linear layer, fp32 math (34 MB). No bf16 build, and no int4: the model has no MatMul | |
| weights for 4-bit MatMulNBits. | |
| **Benchmark.** The lab's voice-conversion set: 4 English recordings (LibriSpeech test-clean) and 2 Chinese ones | |
| (AISHELL-1 test), each converted into the English (LibriSpeech) and the Chinese (AISHELL-1) voice of the lab's TTS | |
| prompts: 12 conversions. Each output is judged by Whisper large-v3-turbo's WER (English sources) or CER (Chinese) | |
| against the source's transcript, WavLM-base-plus-sv speaker similarity to the target voice, and DNSMOS (P.835 | |
| overall). The original is the official code (openvoice @ 74a1d14) in fp32 on CUDA, given the same noise and the same | |
| spectrograms as the graphs. Pass, per source language: error rate at most 2 points above the original's, similarity | |
| at most 0.03 below, DNSMOS at most 0.2 below. Real-time factor (processing time / audio time) on an RTX 3080 Laptop | |
| GPU, whole pipeline. SNR is each output against the original's on the same input and noise. | |
| | | English WER | similarity | DNSMOS | Chinese CER | similarity | DNSMOS | SNR vs original | Real-time factor | | |
| |---|---|---|---|---|---|---|---|---| | |
| | original (PyTorch fp32, CUDA) | 2.6 % | 0.842 | 3.30 | 27.1 % | 0.797 | 2.89 | | 0.05 | | |
| | `fp16/` (CUDA) | 2.6 % | 0.842 | 3.31 | 25.0 % | 0.792 | 2.90 | 23 dB | 0.06 | | |
| | `int8/` (CUDA) | 2.6 % | 0.845 | 3.30 | 25.0 % | 0.789 | 2.89 | 5 dB | 0.04 | | |
| The Chinese error rate is high for the original too: the two AISHELL-1 sources are about 4 s each, and the converter | |
| slurs some characters. The int8 output differs audibly in detail from the original's (low SNR) while scoring the | |
| same; use `fp16/` if you need it close to the original sample by sample. The fp32 graphs, checked before the | |
| ladder, match the original to 5e-6 (converter) and 1e-6 (reference encoder), 39 dB SNR on this set on CUDA. | |
| ## How these were made | |
| From [OpenVoice V2](https://huggingface.co/myshell-ai/OpenVoiceV2) at revision `f36e7ed`. The graphs were exported from the original PyTorch modules with | |
| torch.onnx (opset 17 / 18) and checked against them. Weight-only quantization: int8 is a per-output-channel int8 | |
| weight behind DequantizeLinear, int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was | |
| benchmarked against the original before upload (pass, per language: error rate at most 2 points above the | |
| original's, similarity to the target voice at most 0.03 below, DNSMOS at most 0.2 below), and the `sha256` column above is what | |
| the files must hash to. | |
| ## License | |
| MIT. These files are a derivative of the original model and carry its license: it is copied in this | |
| repository, and it governs your use of these files. The original model card follows unchanged. | |
| --- | |
| # Original model card | |
| # OpenVoice V2 | |
| <a href="https://trendshift.io/repositories/6161" target="_blank"><img src="https://trendshift.io/api/badge/repositories/6161" alt="myshell-ai%2FOpenVoice | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a> | |
| In April 2024, we release OpenVoice V2, which includes all features in V1 and has: | |
| 1. Better Audio Quality. OpenVoice V2 adopts a different training strategy that delivers better audio quality. | |
| 2. Native Multi-lingual Support. English, Spanish, French, Chinese, Japanese and Korean are natively supported in OpenVoice V2. | |
| 3. Free Commercial Use. Starting from April 2024, both V2 and V1 are released under MIT License. Free for commercial use. | |
| <video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/641de0213239b631552713e4/uCHTHD9OUotgOflqDu3QK.mp4"></video> | |
| ### Features | |
| - **Accurate Tone Color Cloning.** OpenVoice can accurately clone the reference tone color and generate speech in multiple languages and accents. | |
| - **Flexible Voice Style Control.** OpenVoice enables granular control over voice styles, such as emotion and accent, as well as other style parameters including rhythm, pauses, and intonation. | |
| - **Zero-shot Cross-lingual Voice Cloning.** Neither of the language of the generated speech nor the language of the reference speech needs to be presented in the massive-speaker multi-lingual training dataset. | |
| ### How to Use | |
| Please see [usage](https://github.com/myshell-ai/OpenVoice/blob/main/docs/USAGE.md) for detailed instructions. | |
| # Usage | |
| ## Table of Content | |
| - [Quick Use](#quick-use): directly use OpenVoice without installation. | |
| - [Linux Install](#linux-install): for researchers and developers only. | |
| - [V1](#openvoice-v1) | |
| - [V2](#openvoice-v2) | |
| - [Install on Other Platforms](#install-on-other-platforms): unofficial installation guide contributed by the community | |
| ## Quick Use | |
| The input speech audio of OpenVoice can be in **Any Language**. OpenVoice can clone the voice in that speech audio, and use the voice to speak in multiple languages. For quick use, we recommend you to try the already deployed services: | |
| - [British English](https://app.myshell.ai/widget/vYjqae) | |
| - [American English](https://app.myshell.ai/widget/nEFFJf) | |
| - [Indian English](https://app.myshell.ai/widget/V3iYze) | |
| - [Australian English](https://app.myshell.ai/widget/fM7JVf) | |
| - [Spanish](https://app.myshell.ai/widget/NNFFVz) | |
| - [French](https://app.myshell.ai/widget/z2uyUz) | |
| - [Chinese](https://app.myshell.ai/widget/fU7nUz) | |
| - [Japanese](https://app.myshell.ai/widget/IfIB3u) | |
| - [Korean](https://app.myshell.ai/widget/q6ZjIn) | |
| ## Linux Install | |
| This section is only for developers and researchers who are familiar with Linux, Python and PyTorch. Clone this repo, and run | |
| ``` | |
| conda create -n openvoice python=3.9 | |
| conda activate openvoice | |
| git clone git@github.com:myshell-ai/OpenVoice.git | |
| cd OpenVoice | |
| pip install -e . | |
| ``` | |
| No matter if you are using V1 or V2, the above installation is the same. | |
| ### OpenVoice V1 | |
| Download the checkpoint from [here](https://myshell-public-repo-host.s3.amazonaws.com/openvoice/checkpoints_1226.zip) and extract it to the `checkpoints` folder. | |
| **1. Flexible Voice Style Control.** | |
| Please see [`demo_part1.ipynb`](https://github.com/myshell-ai/OpenVoice/blob/main/demo_part1.ipynb) for an example usage of how OpenVoice enables flexible style control over the cloned voice. | |
| **2. Cross-Lingual Voice Cloning.** | |
| Please see [`demo_part2.ipynb`](https://github.com/myshell-ai/OpenVoice/blob/main/demo_part2.ipynb) for an example for languages seen or unseen in the MSML training set. | |
| **3. Gradio Demo.**. We provide a minimalist local gradio demo here. We strongly suggest the users to look into `demo_part1.ipynb`, `demo_part2.ipynb` and the [QnA](QA.md) if they run into issues with the gradio demo. Launch a local gradio demo with `python -m openvoice_app --share`. | |
| ### OpenVoice V2 | |
| Download the checkpoint from [here](https://myshell-public-repo-hosting.s3.amazonaws.com/openvoice/checkpoints_v2_0417.zip) and extract it to the `checkpoints_v2` folder. | |
| Install [MeloTTS](https://github.com/myshell-ai/MeloTTS): | |
| ``` | |
| pip install git+https://github.com/myshell-ai/MeloTTS.git | |
| python -m unidic download | |
| ``` | |
| **Demo Usage.** Please see [`demo_part3.ipynb`](https://github.com/myshell-ai/OpenVoice/blob/main/demo_part3.ipynb) for example usage of OpenVoice V2. Now it natively supports English, Spanish, French, Chinese, Japanese and Korean. | |
| ## Install on Other Platforms | |
| This section provides the unofficial installation guides by open-source contributors in the community: | |
| - Windows | |
| - [Guide](https://github.com/Alienpups/OpenVoice/blob/main/docs/USAGE_WINDOWS.md) by [@Alienpups](https://github.com/Alienpups) | |
| - You are welcome to contribute if you have a better installation guide. We will list you here. | |
| - Docker | |
| - [Guide](https://github.com/StevenJSCF/OpenVoice/blob/update-docs/docs/DF_USAGE.md) by [@StevenJSCF](https://github.com/StevenJSCF) | |
| - You are welcome to contribute if you have a better installation guide. We will list you here. | |
| ### Links | |
| - [Github](https://github.com/myshell-ai/OpenVoice) | |
| - [HFDemo](https://huggingface.co/spaces/myshell-ai/OpenVoiceV2) | |
| - [Discord](https://discord.gg/myshell) | |