Instructions to use fdemelo/x-vits with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Piper
How to use fdemelo/x-vits with Piper:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
x-vits · Expresso ex02 (English, expressive)
An expressive Piper/VITS voice that can speak in six styles and stress the words you mark, or the words it thinks matter. Made with x-vits, a recipe that adds prosody modelling (intonation, rhythm, emphasis) to small, fast VITS voices. It's about 25M parameters in all, exported to ONNX, and runs on a CPU.
What's different from a plain Piper voice
- 🎭 Styles:
default,happy,sad,confused,enunciated,laughing. - 💥 Emphasis: write
*word*to stress a word. Without marks, a small model predicts which words to stress. - 🌊 Sampled intonation: a flow-matching model picks a plausible pitch, voicing and energy contour per phoneme, instead of averaging every sentence into the same melody.
- 🎚️ Style-aware timing: sad slows down, enunciated takes its pauses.
🎧 Samples
About the comparison. "Piper" below is not a flat, neutral voice. It's the same speaker, fine-tuned with plain Piper/VITS on the same expressive recordings in all six styles, so it already has lively prosody: VITS samples its own timing and intonation, and this one learned an average of the six speaking manners. What it can't do is choose: there is no way to ask it for "sad" or to stress a particular word. x-vits adds that control, and gets closer to how the speaker actually read each sentence (pitch contour correlation with the recording 0.44 vs 0.35; per-style timing within about 10–14 % of the recordings instead of up to 24 % off).
One sentence, six styles
The clearest difference: plain Piper has a single way to read "You broke the tie score!". x-vits reads it in whichever style you ask for. The speaker's own recordings of this held-out sentence are on the right.
| Piper (any style): | ||
|---|---|---|
| style | x-vits | recording |
| default | ||
| happy | ||
| sad | ||
| confused | ||
| enunciated | ||
| laughing | ||
Measured on these clips, x-vits moves the same way as the speaker: happy is higher (303 vs 253 Hz for default; recording 338), sad is narrowest (pitch range 6.7 semitones; recording 5.0), confused is widest and slower (17.2 semitones; recording 18.4), and enunciated is the slowest. Laughing is the weak spot: the voice can't produce laughter, so it stays low-key where the speaker breaks into a laugh.
Different sentences, each in its recorded style
Held-out sentences, never seen in training, each with the recording of that sentence in that style for reference. All x-vits and Piper samples use the same random seed and none were cherry-picked.
| style | text | Piper | x-vits | recording |
|---|---|---|---|---|
| default | I'm more of an all terrain vehicle type. | |||
| happy | We didn't get the weather right, guys. | |||
| sad | No, in Salt Lake City, today, it won't snow. | |||
| confused | Is young adult fiction very popular? | |||
| enunciated | One PM to nine PM, it'll be cloudy, and in the low fifties. | |||
| laughing | Their moment had come, or so they said. |
Emphasis, by moving the *mark* (style default)
Piper has no equivalent: it stresses whatever its training suggests.
| text | x-vits |
|---|---|
| I never said she stole my money. | |
| *I* never said she stole my money. | |
| I never said *she* stole my money. | |
| I never said she stole my *money*. |
🚀 Usage
sudo apt install libespeak-ng1 # phonemizer backend (macOS: brew install espeak-ng)
pip install "x-vits @ git+https://github.com/femelo/x-vits"
import soundfile as sf
from xvits.runtime import XVits
voice = XVits.from_pretrained("fdemelo/x-vits")
audio = voice.synthesize("We didn't get the weather right, guys.", style="happy")
sf.write("happy.wav", audio, voice.sample_rate)
# explicit emphasis, a different style, a fixed seed and a faster CPU setting
audio = voice.synthesize("I never said *she* stole my money.", style="confused", seed=7, steps=4)
sf.write("confused.wav", audio, voice.sample_rate)
Or from the command line:
xvits-speak --model fdemelo/x-vits --text "Is it *really* you?" --style sad --out sad.wav
| option | values | |
|---|---|---|
style |
default happy sad confused enunciated laughing |
none gives a neutral mix |
emphasis |
auto · marked · none |
auto: marks if present, otherwise predicted |
emphasis_scale |
default 1.5 | 1.0 = average emphasis in the recordings |
length_scale |
default 1.08 | larger is slower |
steps |
default 16 | 4 is about 2.5× faster on CPU and changes pitch by about 0.1 semitone |
seed |
int | reproducible output |
Speed with onnxruntime on 4 CPU threads: real-time factor 0.60 (16 steps) or 0.24 (4 steps).
📦 Files
| file | role | size |
|---|---|---|
encoder.onnx |
phonemes → text states | 26 MB |
predictor.onnx |
one step of the flow-matching prosody predictor (style-conditioned) | 31 MB |
durations.onnx |
style-aware stochastic duration predictor | 2.5 MB |
acoustic.onnx |
prosody-conditioned prior, length regulation, flow, HiFi-GAN | 36 MB |
prominence.onnx |
predicts which words to stress | 4.8 MB |
dp_cond.npz, emphasis_template.json |
style vectors for the duration predictor; the emphasis shape | small |
runtime.json, config.json |
runtime settings; Piper phoneme map (eSpeak en) |
These graphs are driven by xvits.runtime, so they aren't drop-in files for Piper's own runtime.
The export was verified against PyTorch end to end (maximum difference about 1e-4 at several sentence lengths).
🧪 Training
- Base: Piper's LJSpeech
mediumcheckpoint (LJSpeech is public domain). - Data: Expresso speaker ex02, read speech in six styles,
2.2 h after leaving out whisper, singing, long-form and
<laugh>rows. 245 utterances held out. - Recipe (x-vits):
- fine-tune the base checkpoint on ex02;
- extract per-phoneme duration, pitch and energy targets;
- fine-tune with a zero-initialised prosody input to the prior, including frame-level interpolation;
- train an OT-CFM prosody predictor with a style input;
- train a style-conditioned duration predictor;
- train a word-prominence student on acoustic labels;
- fit an emphasis template anchored on the stressed vowel, from Expresso's
*emphasis*marks.
Held-out results (pitch match is the correlation of the pitch contour with the recording of the same sentence):
| pitch match | WER (Whisper) | UTMOS | |
|---|---|---|---|
| Piper, fine-tuned | 0.35 | 1.2 % | 4.08 |
| x-vits | 0.44 | 0.6 % | 4.04 |
On marked words, the pitch peak rises from +2.3 to +4.6 semitones above the sentence median (the recordings: +5.3). The prominence student finds the recording's main stress 42 % of the time, against 34 % for "last word" and 15 % for a random word.
⚠️ License and limitations
- CC BY-NC 4.0. Expresso is licensed for non-commercial use only, and so is this voice. The x-vits code is MIT.
- English, one female speaker, read speech. Styles are as recorded in Expresso; there are no other emotions.
- Laughing is rendered through intonation and timing; the voice doesn't produce actual laughter.
- Questions sometimes end with a smaller rise than a natural reading.
- Don't use this voice to impersonate real people or to mislead listeners.
Citation
If you use this voice, please cite Expresso:
@inproceedings{nguyen2023expresso,
title = {EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis},
author = {Nguyen, Tu Anh and Hsu, Wei-Ning and D'Avirro, Antony and Shi, Bowen and Gat, Itai and
Fazel-Zarani, Maryam and Remez, Tal and Copet, Jade and Synnaeve, Gabriel and Hassid, Michael and
Kreuk, Felix and Adi, Yossi and Dupoux, Emmanuel},
booktitle = {Interspeech},
year = {2023}
}
- Downloads last month
- -