File size: 13,317 Bytes
15627e7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1be2e31
 
 
 
15627e7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
---
base_model: myshell-ai/OpenVoiceV2
base_model_relation: quantized
license: mit
pipeline_tag: audio-to-audio
library_name: onnx
tags:
- onnx
- moonbox
- onnxruntime
- openvoice
- voice-conversion
- tone-color-converter
- multilingual
---

# OpenVoice V2 ONNX

ONNX builds of [OpenVoice V2](https://huggingface.co/myshell-ai/OpenVoiceV2), made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each
precision is one folder holding the graphs and a small Python `pipeline.py` that does the rest (features,
resampling, the conversion steps); what the graphs take and return is under *How to run*.

## Files

The last column is the benchmark on Moonbox's voice-conversion set (English LibriSpeech and Chinese AISHELL-1
recordings, each converted into an English and a Chinese target voice; details under *How to run*): error rate of a
Whisper transcript of the output against the source's transcript, speaker similarity to the target voice and DNSMOS,
against the original PyTorch model.

| Precision | File | Size | sha256 | Benchmark |
|---|---|---|---|---|
| fp16 | [`fp16/config.json`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/fp16/config.json) | 0.8 kB | `9dfff60350b8` | WER 2.6 % / CER 25.0 % (original 2.6 / 27.1, fp32), similarity 0.842 / 0.792 (0.842 / 0.797), DNSMOS 3.31 / 2.90 (3.30 / 2.89); SNR 23 dB vs original; RTF 0.06 (CUDA) |
| fp16 | [`fp16/converter.onnx`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/fp16/converter.onnx) | 64.3 MB | `22a79ab872d1` | WER 2.6 % / CER 25.0 % (original 2.6 / 27.1, fp32), similarity 0.842 / 0.792 (0.842 / 0.797), DNSMOS 3.31 / 2.90 (3.30 / 2.89); SNR 23 dB vs original; RTF 0.06 (CUDA) |
| fp16 | [`fp16/pipeline.py`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/fp16/pipeline.py) | 3.7 kB | `876563834920` | WER 2.6 % / CER 25.0 % (original 2.6 / 27.1, fp32), similarity 0.842 / 0.792 (0.842 / 0.797), DNSMOS 3.31 / 2.90 (3.30 / 2.89); SNR 23 dB vs original; RTF 0.06 (CUDA) |
| fp16 | [`fp16/ref_encoder.onnx`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/fp16/ref_encoder.onnx) | 1.6 MB | `50742432520f` | WER 2.6 % / CER 25.0 % (original 2.6 / 27.1, fp32), similarity 0.842 / 0.792 (0.842 / 0.797), DNSMOS 3.31 / 2.90 (3.30 / 2.89); SNR 23 dB vs original; RTF 0.06 (CUDA) |
| int8 | [`int8/config.json`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/int8/config.json) | 0.8 kB | `9dfff60350b8` | WER 2.6 % / CER 25.0 %, similarity 0.845 / 0.789, DNSMOS 3.30 / 2.89; SNR 5 dB vs original; RTF 0.04 (CUDA) |
| int8 | [`int8/converter.onnx`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/int8/converter.onnx) | 32.7 MB | `17c254cb33ca` | WER 2.6 % / CER 25.0 %, similarity 0.845 / 0.789, DNSMOS 3.30 / 2.89; SNR 5 dB vs original; RTF 0.04 (CUDA) |
| int8 | [`int8/pipeline.py`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/int8/pipeline.py) | 3.7 kB | `876563834920` | WER 2.6 % / CER 25.0 %, similarity 0.845 / 0.789, DNSMOS 3.30 / 2.89; SNR 5 dB vs original; RTF 0.04 (CUDA) |
| int8 | [`int8/ref_encoder.onnx`](https://huggingface.co/moonbox-dev/OpenVoiceV2-ONNX/blob/main/int8/ref_encoder.onnx) | 0.8 MB | `2e8b8b34c880` | WER 2.6 % / CER 25.0 %, similarity 0.845 / 0.789, DNSMOS 3.30 / 2.89; SNR 5 dB vs original; RTF 0.04 (CUDA) |

## How to run

OpenVoice V2's tone-colour converter (MyShell: re-voices a recording into the voice of a reference recording,
keeping its words and timing; any language, 22.05 kHz output) as two ONNX graphs plus `pipeline.py`, a numpy runtime
that reproduces the official `ToneColorConverter.extract_se` / `convert(tau=0.3)` around them. The base speakers
(MeloTTS) are not part of this repository: give it any recording as the source.

- `ref_encoder.onnx`: linear spectrogram `spec` [1, T, 513] (22.05 kHz, n_fft 1024, hop 256) -> tone-colour
  embedding `se` [1, 256].
- `converter.onnx`: the source's spectrogram `spec` [1, 513, T], the posterior encoder's sampling noise `noise`
  [1, 192, T], and the source and target embeddings `src_se`, `tgt_se` [1, 256, 1] -> audio `wav` [1, 1, 256 T]
  (the original `SynthesizerTrn.voice_conversion` with tau 0.3).

The noise is an input, so a seed gives the same output every run. `pipeline.py` computes the spectrogram in numpy
(the original's `spectrogram_torch(center=False)`), resamples with soxr, and embeds the source and the target voice
each in one piece (the official `se_extractor.get_se` first splits the reference with a VAD and averages; for a long
reference you can do the same with `embed` and average the embeddings). The watermark the official code can add is
not included. It needs numpy, onnxruntime (or onnxruntime-gpu), soundfile and soxr:

```bash
python fp16/pipeline.py source.wav voice.wav out.wav --cuda --seed 0
```

```python
import sys, soundfile as sf
sys.path.insert(0, "fp16")
import pipeline
vc = pipeline.OpenVoiceV2("fp16", "cuda")
src, sr = sf.read("source.wav", dtype="float32")
ref, ref_sr = sf.read("voice.wav", dtype="float32")
wav, out_sr = vc(src, sr, ref, ref_sr, seed=0)       # 22.05 kHz
```

**Precisions.** `fp16/`: both graphs fp16 math with fp32 inputs and outputs (66 MB). `int8/`: weight-only int8 for
every convolution, GRU and linear layer, fp32 math (34 MB). No bf16 build, and no int4: the model has no MatMul
weights for 4-bit MatMulNBits.

**Benchmark.** The lab's voice-conversion set: 4 English recordings (LibriSpeech test-clean) and 2 Chinese ones
(AISHELL-1 test), each converted into the English (LibriSpeech) and the Chinese (AISHELL-1) voice of the lab's TTS
prompts: 12 conversions. Each output is judged by Whisper large-v3-turbo's WER (English sources) or CER (Chinese)
against the source's transcript, WavLM-base-plus-sv speaker similarity to the target voice, and DNSMOS (P.835
overall). The original is the official code (openvoice @ 74a1d14) in fp32 on CUDA, given the same noise and the same
spectrograms as the graphs. Pass, per source language: error rate at most 2 points above the original's, similarity
at most 0.03 below, DNSMOS at most 0.2 below. Real-time factor (processing time / audio time) on an RTX 3080 Laptop
GPU, whole pipeline. SNR is each output against the original's on the same input and noise.

| | English WER | similarity | DNSMOS | Chinese CER | similarity | DNSMOS | SNR vs original | Real-time factor |
|---|---|---|---|---|---|---|---|---|
| original (PyTorch fp32, CUDA) | 2.6 % | 0.842 | 3.30 | 27.1 % | 0.797 | 2.89 | | 0.05 |
| `fp16/` (CUDA) | 2.6 % | 0.842 | 3.31 | 25.0 % | 0.792 | 2.90 | 23 dB | 0.06 |
| `int8/` (CUDA) | 2.6 % | 0.845 | 3.30 | 25.0 % | 0.789 | 2.89 | 5 dB | 0.04 |

The Chinese error rate is high for the original too: the two AISHELL-1 sources are about 4 s each, and the converter
slurs some characters. The int8 output differs audibly in detail from the original's (low SNR) while scoring the
same; use `fp16/` if you need it close to the original sample by sample. The fp32 graphs, checked before the
ladder, match the original to 5e-6 (converter) and 1e-6 (reference encoder), 39 dB SNR on this set on CUDA.

## How these were made

From [OpenVoice V2](https://huggingface.co/myshell-ai/OpenVoiceV2) at revision `f36e7ed`. The graphs were exported from the original PyTorch modules with
torch.onnx (opset 17 / 18) and checked against them. Weight-only quantization: int8 is a per-output-channel int8
weight behind DequantizeLinear, int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was
benchmarked against the original before upload (pass, per language: error rate at most 2 points above the
original's, similarity to the target voice at most 0.03 below, DNSMOS at most 0.2 below), and the `sha256` column above is what
the files must hash to.

## License

MIT. These files are a derivative of the original model and carry its license: it is copied in this
repository, and it governs your use of these files. The original model card follows unchanged.

---

# Original model card

# OpenVoice V2

<a href="https://trendshift.io/repositories/6161" target="_blank"><img src="https://trendshift.io/api/badge/repositories/6161" alt="myshell-ai%2FOpenVoice | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>


In April 2024, we release OpenVoice V2, which includes all features in V1 and has:

1. Better Audio Quality. OpenVoice V2 adopts a different training strategy that delivers better audio quality.

2. Native Multi-lingual Support. English, Spanish, French, Chinese, Japanese and Korean are natively supported in OpenVoice V2.

3. Free Commercial Use. Starting from April 2024, both V2 and V1 are released under MIT License. Free for commercial use.


<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/641de0213239b631552713e4/uCHTHD9OUotgOflqDu3QK.mp4"></video>

### Features
- **Accurate Tone Color Cloning.** OpenVoice can accurately clone the reference tone color and generate speech in multiple languages and accents.
- **Flexible Voice Style Control.** OpenVoice enables granular control over voice styles, such as emotion and accent, as well as other style parameters including rhythm, pauses, and intonation.
- **Zero-shot Cross-lingual Voice Cloning.** Neither of the language of the generated speech nor the language of the reference speech needs to be presented in the massive-speaker multi-lingual training dataset.

### How to Use
Please see [usage](https://github.com/myshell-ai/OpenVoice/blob/main/docs/USAGE.md) for detailed instructions.

# Usage

## Table of Content

- [Quick Use](#quick-use): directly use OpenVoice without installation.
- [Linux Install](#linux-install): for researchers and developers only.
    - [V1](#openvoice-v1)
    - [V2](#openvoice-v2)
- [Install on Other Platforms](#install-on-other-platforms): unofficial installation guide contributed by the community

## Quick Use

The input speech audio of OpenVoice can be in **Any Language**. OpenVoice can clone the voice in that speech audio, and use the voice to speak in multiple languages. For quick use, we recommend you to try the already deployed services:

- [British English](https://app.myshell.ai/widget/vYjqae)
- [American English](https://app.myshell.ai/widget/nEFFJf)
- [Indian English](https://app.myshell.ai/widget/V3iYze)
- [Australian English](https://app.myshell.ai/widget/fM7JVf)
- [Spanish](https://app.myshell.ai/widget/NNFFVz)
- [French](https://app.myshell.ai/widget/z2uyUz)
- [Chinese](https://app.myshell.ai/widget/fU7nUz)
- [Japanese](https://app.myshell.ai/widget/IfIB3u)
- [Korean](https://app.myshell.ai/widget/q6ZjIn)

## Linux Install

This section is only for developers and researchers who are familiar with Linux, Python and PyTorch. Clone this repo, and run

```
conda create -n openvoice python=3.9
conda activate openvoice
git clone git@github.com:myshell-ai/OpenVoice.git
cd OpenVoice
pip install -e .
```

No matter if you are using V1 or V2, the above installation is the same.

### OpenVoice V1

Download the checkpoint from [here](https://myshell-public-repo-host.s3.amazonaws.com/openvoice/checkpoints_1226.zip) and extract it to the `checkpoints` folder.

**1. Flexible Voice Style Control.**
Please see [`demo_part1.ipynb`](https://github.com/myshell-ai/OpenVoice/blob/main/demo_part1.ipynb) for an example usage of how OpenVoice enables flexible style control over the cloned voice.

**2. Cross-Lingual Voice Cloning.**
Please see [`demo_part2.ipynb`](https://github.com/myshell-ai/OpenVoice/blob/main/demo_part2.ipynb) for an example for languages seen or unseen in the MSML training set.

**3. Gradio Demo.**. We provide a minimalist local gradio demo here. We strongly suggest the users to look into `demo_part1.ipynb`, `demo_part2.ipynb` and the [QnA](QA.md) if they run into issues with the gradio demo. Launch a local gradio demo with `python -m openvoice_app --share`.

### OpenVoice V2

Download the checkpoint from [here](https://myshell-public-repo-hosting.s3.amazonaws.com/openvoice/checkpoints_v2_0417.zip) and extract it to the `checkpoints_v2` folder.

Install [MeloTTS](https://github.com/myshell-ai/MeloTTS):
```
pip install git+https://github.com/myshell-ai/MeloTTS.git
python -m unidic download
```

**Demo Usage.** Please see [`demo_part3.ipynb`](https://github.com/myshell-ai/OpenVoice/blob/main/demo_part3.ipynb) for example usage of OpenVoice V2. Now it natively supports English, Spanish, French, Chinese, Japanese and Korean.


## Install on Other Platforms

This section provides the unofficial installation guides by open-source contributors in the community:

- Windows
  - [Guide](https://github.com/Alienpups/OpenVoice/blob/main/docs/USAGE_WINDOWS.md) by [@Alienpups](https://github.com/Alienpups)
  - You are welcome to contribute if you have a better installation guide. We will list you here.
- Docker
  - [Guide](https://github.com/StevenJSCF/OpenVoice/blob/update-docs/docs/DF_USAGE.md) by [@StevenJSCF](https://github.com/StevenJSCF)
  - You are welcome to contribute if you have a better installation guide. We will list you here.


### Links
- [Github](https://github.com/myshell-ai/OpenVoice)
- [HFDemo](https://huggingface.co/spaces/myshell-ai/OpenVoiceV2)
- [Discord](https://discord.gg/myshell)