x-vits · Expresso ex02 (English, expressive)

An expressive Piper/VITS voice that can speak in six styles and stress the words you mark, or the words it thinks matter. Made with x-vits, a recipe that adds prosody modelling (intonation, rhythm, emphasis) to small, fast VITS voices. It's about 25M parameters in all, exported to ONNX, and runs on a CPU.

What's different from a plain Piper voice

  • 🎭 Styles: default, happy, sad, confused, enunciated, laughing.
  • 💥 Emphasis: write *word* to stress a word. Without marks, a small model predicts which words to stress.
  • 🌊 Sampled intonation: a flow-matching model picks a plausible pitch, voicing and energy contour per phoneme, instead of averaging every sentence into the same melody.
  • 🎚️ Style-aware timing: sad slows down, enunciated takes its pauses.

🎧 Samples

About the comparison. "Piper" below is not a flat, neutral voice. It's the same speaker, fine-tuned with plain Piper/VITS on the same expressive recordings in all six styles, so it already has lively prosody: VITS samples its own timing and intonation, and this one learned an average of the six speaking manners. What it can't do is choose: there is no way to ask it for "sad" or to stress a particular word. x-vits adds that control, and gets closer to how the speaker actually read each sentence (pitch contour correlation with the recording 0.44 vs 0.35; per-style timing within about 10–14 % of the recordings instead of up to 24 % off).

One sentence, six styles

The clearest difference: plain Piper has a single way to read "You broke the tie score!". x-vits reads it in whichever style you ask for. The speaker's own recordings of this held-out sentence are on the right.

Piper (any style):
stylex-vitsrecording
default
happy
sad
confused
enunciated
laughing

Measured on these clips, x-vits moves the same way as the speaker: happy is higher (303 vs 253 Hz for default; recording 338), sad is narrowest (pitch range 6.7 semitones; recording 5.0), confused is widest and slower (17.2 semitones; recording 18.4), and enunciated is the slowest. Laughing is the weak spot: the voice can't produce laughter, so it stays low-key where the speaker breaks into a laugh.

Different sentences, each in its recorded style

Held-out sentences, never seen in training, each with the recording of that sentence in that style for reference. All x-vits and Piper samples use the same random seed and none were cherry-picked.

styletextPiperx-vitsrecording
defaultI'm more of an all terrain vehicle type.
happyWe didn't get the weather right, guys.
sadNo, in Salt Lake City, today, it won't snow.
confusedIs young adult fiction very popular?
enunciatedOne PM to nine PM, it'll be cloudy, and in the low fifties.
laughingTheir moment had come, or so they said.

Emphasis, by moving the *mark* (style default)

Piper has no equivalent: it stresses whatever its training suggests.

textx-vits
I never said she stole my money.
*I* never said she stole my money.
I never said *she* stole my money.
I never said she stole my *money*.

🚀 Usage

sudo apt install libespeak-ng1            # phonemizer backend (macOS: brew install espeak-ng)
pip install "x-vits @ git+https://github.com/femelo/x-vits"
import soundfile as sf
from xvits.runtime import XVits

voice = XVits.from_pretrained("fdemelo/x-vits")

audio = voice.synthesize("We didn't get the weather right, guys.", style="happy")
sf.write("happy.wav", audio, voice.sample_rate)

# explicit emphasis, a different style, a fixed seed and a faster CPU setting
audio = voice.synthesize("I never said *she* stole my money.", style="confused", seed=7, steps=4)
sf.write("confused.wav", audio, voice.sample_rate)

Or from the command line:

xvits-speak --model fdemelo/x-vits --text "Is it *really* you?" --style sad --out sad.wav
option values
style default happy sad confused enunciated laughing none gives a neutral mix
emphasis auto · marked · none auto: marks if present, otherwise predicted
emphasis_scale default 1.5 1.0 = average emphasis in the recordings
length_scale default 1.08 larger is slower
steps default 16 4 is about 2.5× faster on CPU and changes pitch by about 0.1 semitone
seed int reproducible output

Speed with onnxruntime on 4 CPU threads: real-time factor 0.60 (16 steps) or 0.24 (4 steps).

📦 Files

file role size
encoder.onnx phonemes → text states 26 MB
predictor.onnx one step of the flow-matching prosody predictor (style-conditioned) 31 MB
durations.onnx style-aware stochastic duration predictor 2.5 MB
acoustic.onnx prosody-conditioned prior, length regulation, flow, HiFi-GAN 36 MB
prominence.onnx predicts which words to stress 4.8 MB
dp_cond.npz, emphasis_template.json style vectors for the duration predictor; the emphasis shape small
runtime.json, config.json runtime settings; Piper phoneme map (eSpeak en)

These graphs are driven by xvits.runtime, so they aren't drop-in files for Piper's own runtime. The export was verified against PyTorch end to end (maximum difference about 1e-4 at several sentence lengths).

🧪 Training

  • Base: Piper's LJSpeech medium checkpoint (LJSpeech is public domain).
  • Data: Expresso speaker ex02, read speech in six styles, 2.2 h after leaving out whisper, singing, long-form and <laugh> rows. 245 utterances held out.
  • Recipe (x-vits):
    1. fine-tune the base checkpoint on ex02;
    2. extract per-phoneme duration, pitch and energy targets;
    3. fine-tune with a zero-initialised prosody input to the prior, including frame-level interpolation;
    4. train an OT-CFM prosody predictor with a style input;
    5. train a style-conditioned duration predictor;
    6. train a word-prominence student on acoustic labels;
    7. fit an emphasis template anchored on the stressed vowel, from Expresso's *emphasis* marks.

Held-out results (pitch match is the correlation of the pitch contour with the recording of the same sentence):

pitch match WER (Whisper) UTMOS
Piper, fine-tuned 0.35 1.2 % 4.08
x-vits 0.44 0.6 % 4.04

On marked words, the pitch peak rises from +2.3 to +4.6 semitones above the sentence median (the recordings: +5.3). The prominence student finds the recording's main stress 42 % of the time, against 34 % for "last word" and 15 % for a random word.

⚠️ License and limitations

  • CC BY-NC 4.0. Expresso is licensed for non-commercial use only, and so is this voice. The x-vits code is MIT.
  • English, one female speaker, read speech. Styles are as recorded in Expresso; there are no other emotions.
  • Laughing is rendered through intonation and timing; the voice doesn't produce actual laughter.
  • Questions sometimes end with a smaller rise than a natural reading.
  • Don't use this voice to impersonate real people or to mislead listeners.

Citation

If you use this voice, please cite Expresso:

@inproceedings{nguyen2023expresso,
  title     = {EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis},
  author    = {Nguyen, Tu Anh and Hsu, Wei-Ning and D'Avirro, Antony and Shi, Bowen and Gat, Itai and
               Fazel-Zarani, Maryam and Remez, Tal and Copet, Jade and Synnaeve, Gabriel and Hassid, Michael and
               Kreuk, Felix and Adi, Yossi and Dupoux, Emmanuel},
  booktitle = {Interspeech},
  year      = {2023}
}
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train fdemelo/x-vits