Hinotsuba/OpenVoice-ONNX-v2

OpenVoice V2 ONNX Models

This repository provides the speaker encoder and tone color converter from OpenVoice V2 in ONNX format for local voice cloning and voice conversion. The models can run through ONNX Runtime in C#, C++, Python, Rust, and other supported environments without a PyTorch runtime.

The original architecture and model weights were developed by MyShell and the OpenVoice contributors. See the upstream OpenVoice project.

The converter takes speech audio and changes its voice characteristics using source and target speaker embeddings. A separate text-to-speech system supplies the base speech when generating audio from text.

📦 Repository Contents

File	Purpose
tone_extract.onnx	Extracts a 256-dimensional speaker embedding from a reference spectrogram.
tone_color.onnx	Converts a source spectrogram using source and target speaker embeddings.
tone_config.json	Audio preprocessing parameters and OpenVoice V2 network configuration.
LICENSE	Original MIT license and copyright notice.

Download files using:

https://huggingface.co/Hinotsuba/OpenVoice-ONNX-v2/resolve/main/{filename}

Use a specific repository revision in place of main when pinning a deployment.

🛠️ Tensor Specifications

All floating-point tensors use float32. Both graphs have a fixed batch size of 1. frames denotes the number of spectrogram frames, not waveform samples.

Speaker Encoder: tone_extract.onnx

Direction	Name	Type	Shape	Meaning
Input	input	float32	[1, frames, 513]	Reference audio magnitude spectrogram, with time before frequency.
Output	tone_embedding	float32	[1, 256]	Speaker embedding.

Reshape the extracted embedding to [1, 256, 1] before passing it to the converter.

Tone Color Converter: tone_color.onnx

Direction	Name	Type	Shape	Meaning
Input	audio	float32	[1, 513, frames]	Source audio magnitude spectrogram, with frequency before time.
Input	audio_length	int64	[1]	Number of source spectrogram frames.
Input	src_tone	float32	[1, 256, 1]	Embedding of the source or base voice.
Input	dest_tone	float32	[1, 256, 1]	Embedding of the target voice.
Input	tau	float32	[1]	Explicit latent perturbation scale.
Output	converted_audio	float32	[1, 1, length]	Mono waveform at 22050 Hz; the first two axes are batch and channel.

For a single unpadded input, set audio_length[0] to frames. Read converted_audio[0, 0, :] to obtain the one-dimensional waveform. These samples are audio data; encoding them as WAV or another format is the caller's responsibility.

Audio Preparation

Prepare the spectrogram outside the ONNX graphs. The same preprocessing applies to reference audio and source audio; only the tensor axis order differs.

Decode and resample to mono float PCM at 22050 Hz, using the conventional PCM amplitude scale of approximately [-1, 1]. Then match the upstream spectrogram_torch implementation:

Parameter	Value
FFT size (n_fft / filter_length)	1024
Hop length	256
Window length	1024
Window	Periodic Hann
Padding before STFT	Reflect padding, 384 samples on each side
STFT centering	false
FFT normalization	false
Frequency bins	One-sided, 513 bins
Spectrum values	Linear magnitude: sqrt(real * real + imag * imag + 1e-6)

Use a library with these settings or reproduce the same window, padding, FFT scaling, and magnitude calculation. The graphs expect this linear spectrum, with no mel projection or logarithmic scaling.

🚀 Inference Workflow

1. Obtain clean reference speech from the source voice and the target speaker. For a TTS source, use a reference clip from the same base voice that will generate the speech.
2. Prepare each reference spectrogram and run tone_extract.onnx. Reshape the results to [1, 256, 1] for src_tone and dest_tone.
3. Generate or load the source speech. Prepare its spectrogram as [1, 513, frames].
4. Run tone_color.onnx with the source spectrogram, its frame count, both embeddings, and an explicit tau value.
5. Read the mono waveform from converted_audio and encode or play it at 22050 Hz. Resample separately if the output device needs another rate.

These models are used by Tsubaki TTS Engine for local C#/.NET text-to-speech with Piper synthesis, OpenVoice V2 voice cloning, and audio streaming without a Python runtime.

Export Behavior and Limits

Temperature: tau

tau is a required ONNX input and has no automatic default. In the upstream Python implementation, the internal voice_conversion() method defaults to 1.0, while the public ToneColorConverter.convert() method defaults to 0.3.

This export contains a fixed noise tensor. tau scales the latent perturbation drawn from that tensor: lower values reduce the perturbation, and 0 removes its contribution. Repeated CPU calls with identical inputs and tau values produced identical output in verification. Fresh noise is not sampled on each call.

Input Length

The converter's fixed noise tensor has shape [1, 192, 32768], limiting a single conversion to 32768 spectrogram frames despite the dynamic time axis. At a hop length of 256 and output rate of 22050 Hz, this corresponds to approximately 380 seconds of output audio. Split longer audio into shorter chunks before conversion.

Runtime and Export Metadata

Model	ONNX opset	ONNX IR version	Export producer metadata
tone_extract.onnx	11	6	PyTorch 2.1.2
tone_color.onnx	17	8	PyTorch 2.1.2

CPU inference was verified with ONNX Runtime 1.29.0, including dynamic frame counts and explicit tau values. Execution-provider support and performance depend on the runtime, hardware, and application.

The export script and exact upstream checkpoint revision are not included in this package. The metadata above describes the distributed ONNX files.

⚖️ License and Attribution

The original OpenVoice V2 architecture and weights are released by MyShell and the OpenVoice contributors under the MIT License. The ONNX model files in this repository are distributed under the same terms.

Commercial and non-commercial use are permitted under the MIT License. Preserve the copyright and permission notice when redistributing the files. See LICENSE for the full original notice, including Copyright 2024 MyShell.ai.

Upstream project: myshell-ai/OpenVoice.