EmoFI-TTS

Pretrained weights for EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation (arXiv:2610.30477).

A face image sets the emotion and its intensity, a reference utterance sets the speaker identity, and text sets the content. The model is a FastSpeech2 / Meta-StyleSpeech backbone with a single linearly projected style vector, followed by an intensity-conditioned refinement module and a HiFi-GAN vocoder. Code, training recipes, and evaluation scripts: github.com/luelhagos/Emofi.

Files

File	What it is	Place at (repo root)
refinement_15000.pth.tar	Released full system: backbone + refinement module	output/ckpt/MEAD_intensity/15000.pth.tar
backbone_30000.pth.tar	Backbone after the cross-speaker stage; starting point for refinement training	output/ckpt/MEAD/30000.pth.tar
warmstart.pth.tar	Backbone before the cross-speaker stage; starting point for jobs/train.sh	output/ckpt/pretrained/warmstart.pth.tar
generator_universal.pth.tar	HiFi-GAN universal vocoder	hifigan/
face_encoder.pt	Face encoder (ResNet50 + MLP, 768-d), needed only to embed new face images	face_encoder/
best_model.pt	Wav2Vec2 speech-emotion judge used for emotion and intensity evaluation	ser_judge/

Download everything with:

hf download luel/emofi-tts --local-dir .

Training and evaluation on MEAD also need the preprocessed data in luel/emofi-mead (gated, MEAD research license).

Results (MEAD test set, image-only condition)

	UTMOS	WER	Emotion WA	Intensity WA	L1 / L2 / L3 recall
EmoFI-TTS	3.14	0.069	0.717	0.654	0.764 / 0.428 / 0.741
Real recordings	2.86	0.066	0.730	0.622	0.697 / 0.346 / 0.803

Emotion and intensity are scored by the released SER judge; see the paper for the full protocol and baselines.

Intended use and limitations

Research use. Trained on MEAD only: six emotions (neutral, angry, happy, sad, surprised, fear) at three intensity levels, English, 46 MEAD speakers in a studio setting, with a speaker-disjoint test split. Voice transfer to speakers and recording conditions outside MEAD is not evaluated and should not be assumed. Do not use to imitate a person's voice without consent.

License and credits

Code and the TTS checkpoints are released under the MIT License. The HiFi-GAN universal generator is from the HiFi-GAN authors (MIT). The face encoder and the speech-emotion judge were developed by Yejin Jeon and are included with permission.

Citation

@article{jeon2026emofi,
  title   = {EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation},
  author  = {Jeon, Yejin and Beyene, Luel H. and Maltais, Marie and Kim, Hyounghun and Adelani, David I.},
  journal = {arXiv preprint arXiv:2610.30477},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.30477}
}