--- language: - en - es - fr - de - it - pt - pl - tr - ru - nl - cs - ar - zh - ja - hu - ko - hi tags: - tts - onnx - xtts - xttsv2 - voice clone - streaming - gpt2 - hifigan - multilingual - vq - perceiver encoder license: apache-2.0 base_model: coqui/XTTS-v2 --- # XTTSv2 Streaming ONNX **Streaming text-to-speech inference for [XTTSv2](https://arxiv.org/abs/2406.04904) using ONNX Runtime — no PyTorch required.** This repository provides a complete, CPU-friendly, streaming TTS pipeline built on ONNX-exported XTTSv2 models. It replaces the original PyTorch inference path with pure Python/NumPy logic while preserving full compatibility with the XTTSv2 architecture. --- ## Features - **Zero-shot voice cloning** from a short (≤ 6 s) reference audio clip. - **Streaming audio output** — audio chunks are yielded as they are generated, enabling low-latency playback. - **Pure ONNX Runtime + NumPy** — no PyTorch dependency at inference time. - **INT8-quantised GPT model** option for reduced memory footprint and faster CPU inference. - **Cross-fade chunk stitching** for seamless audio across vocoder boundaries. - **Speed control** via linear interpolation of GPT latents. - **Multilingual support** — 17 languages: English (en), Spanish (es), French (fr), German (de), Italian (it), Portuguese (pt), Polish (pl), Turkish (tr), Russian (ru), Dutch (nl), Czech (cs), Arabic (ar), Chinese (zh-cn), Japanese (ja), Hungarian (hu), Korean (ko), Hindi (hi). --- ## Architecture Overview XTTSv2 is composed of four main neural network components, each exported as a separate ONNX model: | Component | ONNX File | Description | |---|---|---| | **Conditioning Encoder** | `conditioning_encoder.onnx` | Six 16-head attention layers + Perceiver Resampler. Compresses a reference mel-spectrogram into 32 × 1024 conditioning latents. | | **Speaker Encoder** | `speaker_encoder.onnx` | H/ASP speaker verification network. Extracts a 512-dim speaker embedding from 16 kHz audio. | | **GPT-2 Decoder** | `gpt_model.onnx` / `gpt_model_int8.onnx` | 30-layer, 1024-dim decoder-only transformer with KV-cache. Autoregressively predicts VQ-VAE audio codes conditioned on text tokens and conditioning latents. | | **HiFi-GAN Vocoder** | `hifigan_vocoder.onnx` | 26M-parameter neural vocoder. Converts GPT-2 hidden states + speaker embedding into a 24 kHz waveform. | Pre-exported embedding tables (text, mel, positional) are stored as `.npy` files in the `embeddings/` directory. ``` ┌─────────────┐ mel @ 22 kHz ┌─────────────────────┐ │ Reference │ ───────────────► │ Conditioning Encoder │──► cond_latents [1,32,1024] │ Audio Clip │ └─────────────────────┘ │ │ audio @ 16 kHz ┌─────────────────────┐ │ │ ───────────────► │ Speaker Encoder │──► speaker_emb [1,512,1] └─────────────┘ └─────────────────────┘ ┌──────────┐ BPE tokens ┌──────────────────────────────────────────┐ │ Text │ ─────────────► │ GPT-2 Decoder (autoregressive + KV$) │──► latents [1,T,1024] └──────────┘ │ prefix = [cond | text+pos | start_mel] │ └──────────────────────────────────────────┘ │ ▼ ┌─────────────────────┐ │ HiFi-GAN Vocoder │──► waveform @ 24 kHz │ (+ speaker_emb) │ └─────────────────────┘ ``` --- ## Repository Structure ``` . ├── README.md # This file ├── requirements.txt # Python dependencies ├── xtts_streaming_pipeline.py # Top-level streaming TTS pipeline ├── xtts_onnx_orchestrator.py # Low-level ONNX AR loop orchestrator ├── xtts_tokenizer.py # BPE tokenizer wrapper ├── zh_num2words.py # Chinese number-to-words utility ├── xtts_onnx/ # ONNX models & assets │ ├── metadata.json # Model architecture metadata │ ├── vocab.json # BPE vocabulary │ ├── mel_stats.npy # Per-channel mel normalisation stats │ ├── conditioning_encoder.onnx # Conditioning encoder │ ├── speaker_encoder.onnx # H/ASP speaker encoder │ ├── gpt_model.onnx # GPT-2 decoder (FP32) │ ├── gpt_model_int8.onnx # GPT-2 decoder (INT8 quantised) │ ├── hifigan_vocoder.onnx # HiFi-GAN vocoder │ └── embeddings/ # Pre-exported embedding tables │ ├── mel_embedding.npy # [1026, 1024] audio code embeddings │ ├── mel_pos_embedding.npy # [608, 1024] mel positional embeddings │ ├── text_embedding.npy # [6681, 1024] BPE text embeddings │ └── text_pos_embedding.npy # [404, 1024] text positional embeddings ├── audio_ref/ # Reference audio clips for voice cloning └── audio_synth/ # Directory for synthesised output ``` --- ## Installation ### Prerequisites - Python ≥ 3.10 - A C compiler may be needed for some dependencies (e.g. `tokenizers`). ### Install dependencies ```bash pip install -r requirements.txt ``` ### Clone from Hugging Face Hub ```bash # Install Git LFS (required for large model files) git lfs install # Clone the repository git clone https://huggingface.co/pltobing/XTTSv2-Streaming-ONNX cd XTTSv2-Streaming-ONNX ``` --- ## Quick Start ### Streaming TTS (command-line) ```bash python -u xtts_streaming_pipeline.py \ --model_dir xtts_onnx/ \ --vocab_path xtts_onnx/vocab.json \ --mel_norms_path xtts_onnx/mel_stats.npy \ --ref_audio audio_ref/male_stewie.mp3 \ --language en \ --output output_streaming.wav ``` ### Python API ```python import numpy as np from xtts_streaming_pipeline import StreamingTTSPipeline # Initialise the pipeline pipeline = StreamingTTSPipeline( model_dir="xtts_onnx/", vocab_path="xtts_onnx/vocab.json", mel_norms_path="xtts_onnx/mel_stats.npy", use_int8_gpt=True, # Use INT8-quantised GPT for faster CPU inference num_threads_gpt=4, # Adjust to your CPU core count ) # Compute speaker conditioning (one-time per speaker) gpt_cond_latent, speaker_embedding = pipeline.get_conditioning_latents( "audio_ref/male_stewie.mp3" ) # Stream synthesis all_chunks = [] for audio_chunk in pipeline.inference_stream( text="Hello, this is a streaming text-to-speech demo.", language="en", gpt_cond_latent=gpt_cond_latent, speaker_embedding=speaker_embedding, stream_chunk_size=20, # AR tokens per vocoder call speed=1.0, # 1.0 = normal speed ): all_chunks.append(audio_chunk) # In a real application, you would play or stream each chunk here. # Concatenate all chunks into a single waveform full_audio = np.concatenate(all_chunks, axis=0) # Save to file import soundfile as sf sf.write("output.wav", full_audio, 24000) ``` --- ## Configuration ### SamplingConfig Control the autoregressive token sampling behaviour: | Parameter | Default | Description | |---|---|---| | `temperature` | `0.75` | Softmax temperature. Lower = more deterministic. | | `top_k` | `50` | Keep only the top-*k* most probable tokens. | | `top_p` | `0.85` | Nucleus sampling cumulative probability threshold. | | `repetition_penalty` | `10.0` | Penalise previously generated tokens. | | `do_sample` | `True` | `True` = multinomial sampling; `False` = greedy argmax. | ```python from xtts_onnx_orchestrator import SamplingConfig sampling = SamplingConfig( temperature=0.65, top_k=30, top_p=0.90, repetition_penalty=10.0, do_sample=True, ) for chunk in pipeline.inference_stream(text, "en", cond, spk, sampling=sampling): ... ``` ### GPTConfig Model architecture parameters are automatically loaded from `metadata.json`. Key fields: | Parameter | Value | Description | |---|---|---| | `n_layer` | 30 | Number of GPT-2 transformer layers | | `embed_dim` | 1024 | Hidden dimension | | `num_heads` | 16 | Number of attention heads | | `head_dim` | 64 | Per-head dimension | | `num_audio_tokens` | 1026 | Audio vocabulary (1024 VQ codes + start + stop) | | `perceiver_output_len` | 32 | Conditioning latent sequence length | | `max_gen_mel_tokens` | 605 | Maximum generated audio tokens | --- ## Module Reference ### `xtts_streaming_pipeline.py` Top-level streaming pipeline. | Class / Function | Description | |---|---| | `StreamingTTSPipeline` | Main pipeline class. Owns sessions, tokenizer, orchestrator. | | `StreamingTTSPipeline.get_conditioning_latents()` | Extract GPT conditioning + speaker embedding from reference audio. | | `StreamingTTSPipeline.inference_stream()` | Generator that yields audio chunks for a text segment. | | `StreamingTTSPipeline.time_scale_gpt_latents_numpy()` | Linearly time-scale GPT latents for speed control. | | `wav_to_mel_cloning_numpy()` | Compute normalised log-mel spectrogram (NumPy, 22 kHz). | | `crossfade_chunks()` | Cross-fade consecutive vocoder waveform chunks. | ### `xtts_onnx_orchestrator.py` Low-level ONNX autoregressive loop. | Class / Function | Description | |---|---| | `ONNXSessionManager` | Loads and manages all ONNX sessions + embedding tables. | | `XTTSOrchestratorONNX` | Drives the GPT-2 AR loop with KV-cache and logits processing. | | `GPTConfig` | Model architecture hyper-parameters (from `metadata.json`). | | `SamplingConfig` | Token sampling hyper-parameters. | | `apply_repetition_penalty()` | NumPy repetition penalty on logits. | | `apply_temperature()` | Temperature scaling on logits. | | `apply_top_k()` | Top-*k* filtering on logits. | | `apply_top_p()` | Nucleus (top-*p*) filtering on logits. | | `numpy_softmax()` | Numerically-stable softmax in NumPy. | | `numpy_multinomial()` | Inverse-CDF multinomial sampling. | --- ## Performance Notes - **`stream_chunk_size`** controls the latency–quality trade-off: smaller values yield audio sooner but run the vocoder more often (on all accumulated latents). - **Thread count** (`num_threads_gpt`) should be tuned to your CPU. Start with the number of physical cores. - First call to `get_conditioning_latents()` is an expensive step (resampling + mel computation + encoder inference). Cache the results for repeated synthesis with the same speaker. --- ## License This project is licensed under the **Apache License 2.0**. See the [LICENSE](LICENSE) file for details. ``` Copyright 2025 Patrick Lumbantobing, Vertox-AI Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 ``` --- ## Acknowledgements - [Coqui AI](https://github.com/coqui-ai/TTS) for the original XTTSv2 model and training recipe. - [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://arxiv.org/abs/2406.04904) (Casanova et al., 2024). - [ONNX Runtime](https://onnxruntime.ai/) for high-performance cross-platform inference.