--- language: - en - zh tags: - onnx - onnxruntime - openvoice - openvoice-v2 - melotts - text-to-speech - voice-conversion - multilingual license: mit pipeline_tag: text-to-speech --- # OpenVoice V2 ONNX (Melo EN/ZH + tone converter) **Private** ONNX exports for [OpenVoice V2](https://github.com/myshell-ai/OpenVoice) style inference: **MeloTTS** base speech (**English and Chinese** supported via separate Melo exports), **tone-color speaker embedding** from a reference clip, and **tone color conversion** to match the reference voice. Built from a local iOS project under `Docs/OpenVoiceV2/` (`convert_to_onnx.py`, `convert_onnx_precision.py`). This is **not** an official MyShell release; it is a derivative export for experimentation and on-device ONNX Runtime workflows. --- ## Supported languages (this export) | Layer | English | Chinese (中文) | Notes | |-------|---------|----------------|-------| | **MeloTTS → ONNX** | `melotts_en.onnx` | `melotts_zh.onnx` | Same tensor I/O contract; export both with `OPENVOICE_MELO_LANGS=EN,ZH` (see below). | | **`tone_color_converter.onnx`** | ✓ | ✓ | **Language-agnostic:** linear spectrogram + `sid_src` / `sid_tgt` embeddings. | | **`tone_color_se_extractor.onnx`** | ✓ | ✓ | **Language-agnostic:** reference mel (fixed time in export) → 256-d embedding. | **Base speaker `.pth` (OpenVoiceV2 layout):** e.g. `base_speakers/ses/en-newest.pth` for English-style base timbre, `base_speakers/ses/zh.pth` for Chinese-style base timbre — pick the file that matches your Melo language when doing full OpenVoice-style cloning. **Other Melo languages (ES / FR / JP / KR):** same `convert_to_onnx.py` mechanism with `OPENVOICE_MELO_LANGS`; filenames follow `melotts_{es|fr|jp|kr}.onnx`. **Hub snapshot:** may contain **only** `melotts_en.onnx` until you re-export and re-upload; once `melotts_zh.onnx` is present under `fp32/` (and derived `fp16/` / `int8/`), the same layout applies. --- ## Exporting English + Chinese Melo ONNX From `Docs/OpenVoiceV2` (after checkpoints + deps are installed): ```bash export OPENVOICE_MELO_LANGS=EN,ZH python convert_to_onnx.py ``` This produces (alongside shared tone-color ONNX) **`melotts_en.onnx`** and **`melotts_zh.onnx`**. Then run `convert_onnx_precision.py` per directory as usual; precision scripts pick up **every** `melotts_*.onnx` in the input folder. --- ## Repository layout | Path | Precision | Contents | |------|-------------|----------| | `fp32/` | FP32 | `tone_color_se_extractor.onnx`, `tone_color_converter.onnx`, and **one or more** `melotts_*.onnx` (e.g. `melotts_en.onnx`, `melotts_zh.onnx`). | | `fp16/` | Mixed | Same **filenames** as `fp32/`. Global FP16 often breaks ORT on large VITS graphs; the script **falls back to FP32** per file when ORT cannot load the graph. **SE extractor** is often the only true FP16 file. | | `int8/` | Dynamic INT8 (ORT) | Same filenames. **Weight-only dynamic quantization** (`MatMul` / `Gemm`, with `MatMul`-only fallback). | --- ## Model I/O (summary) ### Melo ONNX (`melotts_en.onnx`, `melotts_zh.onnx`, …) | Input | Shape (typical) | Dtype | |-------|-----------------|-------| | `phone_ids` | `[1, S]` | int64 | | `phone_lens` | `[1]` | int64 | | `sid` | `[1]` | int64 | | `tones` | `[1, S]` | int64 | | `lang_ids` | `[1, S]` | int64 | | `bert_feat` | `[1, 1024, S]` | float32 | | `ja_bert` | `[1, 768, S]` | float32 | | Output | Shape | Dtype | |--------|-------|-------| | `audio` | `[1, 1, T_wav]` | float32 | Chinese exports use **Melo’s Chinese text pipeline** (e.g. jieba + multilingual BERT) at **training/export** time; ONNX **inputs remain** the same fields as English. ### `tone_color_se_extractor.onnx` | Input | Shape | Dtype | |-------|-------|-------| | `mel` | `[1, 513, 128]` (fixed time in export) | float32 | | Output | Shape | Dtype | |--------|-------|-------| | `speaker_embedding` | `[1, 256]` | float32 | ### `tone_color_converter.onnx` | Input | Shape | Dtype | |-------|-------|-------| | `y` | `[1, 513, T_mel]` | float32 | | `y_lengths` | `[1]` | int64 | | `sid_src` | `[1, 256, 1]` | float32 | | `sid_tgt` | `[1, 256, 1]` | float32 | | Output | Shape | Dtype | |--------|-------|-------| | `audio_out` | `[1, 1, T_audio]` | float32 | Sampling rate and STFT hyperparameters follow **OpenVoice V2 `converter/config.json`** (typically **22050 Hz**, `filter_length=1024`, `hop_length=256`, `win_length=1024`). --- ## Intended use - Research and **on-device** prototyping with **ONNX Runtime** (CPU / optional CoreML EP on Apple platforms). - **English or Chinese** base speech from the matching **`melotts_*.onnx`**, then shared **SE + converter** ONNX. - Pairs with **OpenVoice V2** checkpoints from [myshell-ai/OpenVoiceV2](https://huggingface.co/myshell-ai/OpenVoiceV2) — **not** re-uploaded here. ## Out-of-scope / limitations - **Not** a drop-in replacement for a separate **CoreML** OpenVoice bundle (different tensor contracts in the app). - **No** bundled text frontend: run **Melo-compatible** preprocessing or use `tts_onnx_sentence.py` (`--melo-language EN_NEWEST` or `ZH`, etc.). - **FP16 / INT8:** validate on **your** ORT version and EP. --- ## Evaluation - **Smoke:** `smoke_onnx_openvoice.py`, `smoke_onnx_precision.py` (all `melotts_*.onnx` in the folder). - **Sentence TTS (Python):** `tts_onnx_sentence.py` with `--melo-language` matching the ONNX file. --- ## How to download ```bash pip install huggingface_hub huggingface-cli download aoiandroid/OpenVoiceV2-ONNX-mirror --local-dir ./openvoice_onnx --include "fp32/*" ``` Use a token with **read** access to this private repo. --- ## Citation (upstream) Cite [MyShell OpenVoice](https://github.com/myshell-ai/OpenVoice) and [MeloTTS](https://github.com/myshell-ai/MeloTTS). This Hub repo is a **convenience export** for on-device experimentation. --- ## 日本語サマリー - **Melo ONNX:** 英語（`melotts_en.onnx`）と中国語（`melotts_zh.onnx`）に対応。`OPENVOICE_MELO_LANGS=EN,ZH` で `convert_to_onnx.py` から両方エクスポート可能（I/O 形式は共通）。 - **コンバータ / SE:** 言語非依存。参照音声で声色を寄せる用途。 - **fp32 / fp16 / int8:** ディレクトリ内の **`melotts_*.onnx` をすべて**精度変換の対象に含めます。 - **公式ではありません。** upstream のライセンス・条件に従ってください。