XTTS v2: Optimized for AMD ROCm
XTTS v2 is a zero-shot, multilingual text-to-speech model that clones a target voice from a few seconds of reference audio and then reads arbitrary text in that voice. This repository packages evaluation/inference for zero-shot multilingual text-to-speech using ONNX Runtime, exported and validated for AMD ROCm so it runs efficiently on AMD GPUs and CPUs, as well as AMD Ryzen AI NPUs.
This is based on the implementation of XTTS v2 found here. This repository contains configurations and scripts optimized for AMD® ROCm™ platforms. You can use the XTTS v2 AMD scripts to reproduce results or export with custom configurations. More details on model performance can be found here.
Task Overview
Task: Zero-shot multilingual text-to-speech / voice cloning
Dataset: 5 fixed English test sentences synthesized from one reference voice, scored against the PyTorch baseline (no labeled corpus — this is a fidelity + quality eval, see Accuracy Pipeline)
Output metrics: RTF, PESQ, STOI, CER, SECS
ONNX Runtime note: CPU runs FP32 only. GPU end-to-end synthesis runs FP32 via
ROCMExecutionProvider(the autoregressive GPT decoder needs the dynamic-shape handling that MIGraphX does not provide efficiently); the FP16 GPU speedup is characterized at the vocoder via MIGraphX profiling rather than end-to-end. NPU omits precision entirely (VitisAI auto-quantizes internally).
AMD ROCm Optimization
This model export has been adapted and validated for AMD Instinct™ / Radeon™ GPUs running ROCm, as well as AMD CPUs and AMD Ryzen AI NPUs. Key points:
- Exported/tested with ROCm
7.2. - Validated backends: ONNX Runtime CPU EP, ONNX Runtime ROCM EP (ROCm GPU, with MIGraphX used for GPU profiling), and VitisAI EP (Ryzen AI NPU).
- No code changes required versus the upstream XTTS v2 implementation — only environment/runtime configuration differs.
- CPU fallback path supported for environments without a ROCm-capable GPU.
| Runtime | Precision | Backend | Hardware | Notes |
|---|---|---|---|---|
| ONNX Runtime | FP32 | CPUExecutionProvider | AMD CPU | Only precision supported on CPU |
| ONNX Runtime | FP32 | ROCMExecutionProvider | AMD GPU (e.g. gfx1150) | End-to-end synthesis; conditioning encoder pinned to CPU due to a NaN issue on ROCm 7.2/gfx1150 |
| ONNX Runtime | FP16 (vocoder-only profiling) | MIGraphX | AMD GPU (e.g. gfx1150) | FP16 speedup characterized at the vocoder via profiling, not end-to-end synthesis |
| ONNX Runtime | Auto (VitisAI-quantized) | VitisAI EP | AMD Ryzen AI NPU (e.g. STX) | Graph partitioned between NPU and CPU fallback for unsupported ops |
Getting Started
For setup instructions, evaluation scripts, and custom configuration options, see the XTTS v2 on GitHub.
Model Details
Model Type: Zero-shot multilingual text-to-speech (voice cloning)
Base Model: coqui/XTTS-v2
Model Stats:
- Decomposed into four ONNX sub-models: conditioning encoder, speaker encoder, GPT-2 decoder (run autoregressively with an explicit KV-cache), and a HiFi-GAN vocoder, plus NumPy embedding tables
- GPT decoder: 443M parameters
- Supported languages: en, es, fr, de, it, pt, pl, tr, ru, nl, cs, ar, zh-cn, hu, ko, ja, hi
- Precision tested: FP32 (CPU, GPU); FP16 (GPU vocoder profiling only); Auto/VitisAI-quantized (NPU)
Accuracy Pipeline
Some metrics are higher-is-better (PESQ, STOI, SECS) and some are lower-is-better (RTF, CER). Absolute values are modest here by design: TTS sampling is stochastic (temperature 0.75), and the ONNX↔PyTorch fidelity metrics compare two independently sampled utterances of the same text rather than a deterministic reference, so they capture "same speaker/prosody, plausibly the same words," not sample-exact agreement.
Metrics Explained
| Metric | Description |
|---|---|
| RTF | Real-Time Factor — synthesis time ÷ generated audio duration. Lower is faster; below 1.0 means faster than real time. The only performance metric here; the others are quality. |
| PESQ | Perceptual Evaluation of Speech Quality (–0.5…4.5) of the ONNX clip using the PyTorch clip as reference. Higher means the ONNX port sounds closer to PyTorch. Low absolute values are expected because the two clips are separately sampled, not aligned frame-for-frame. |
| STOI | Short-Time Objective Intelligibility (0…1) between the ONNX and PyTorch clips. Higher means more intelligibility overlap; like PESQ it is depressed by stochastic sampling misalignment, so treat it as a relative sanity signal, not an absolute score. |
| CER | Character Error Rate — the synthesized audio is transcribed by Whisper large-v3 and compared to the input text. Lower is better; it measures whether the words are actually intelligible, independent of the PyTorch baseline. This is the most trustworthy quality number here because it needs no alignment. |
| SECS | Speaker Embedding Cosine Similarity (0…1) between the synthesized voice and the reference voice (via Resemblyzer). Higher means the clone matches the target speaker more closely — this is the voice-cloning quality metric. |
Accuracy Results
5 English sentences, reference voice = JFK sample — filled from results/validation*/validation_report.json; run make eval-fulldataset-<device> to refresh:
| Device | Precision | RTF (PT / ONNX) | PESQ | STOI | CER (PT / ONNX) | SECS (PT / ONNX) |
|---|---|---|---|---|---|---|
| CPU | FP32 | 2.830 / 2.694 | 1.180 | 0.125 | 0.020 / 0.277 | 0.753 / 0.644 |
| GPU | FP32 | 3.158 / 22.514 | 1.087 | 0.097 | 0.152 / 0.823 | 0.771 / 0.589 |
| NPU | Auto | — / 2.77 (mean, ONNX-only) | N/A | N/A | N/A | N/A |
Dig Deeper
Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?
📂 View the full project on GitHub
The GitHub repository includes:
- Setup and prerequisites for ROCm environments
- Scripts for the supported runners
- Additional model variants and datasets
- Benchmarking and reproduction instructions