Add model card

README.md

@@ -0,0 +1,75 @@
+---
+license: mit
+language:
+  - en
+tags:
+  - text-to-speech
+  - visual-tts
+  - speech-synthesis
+  - diffusion
+  - spatial-audio
+pipeline_tag: text-to-speech
+---
+
+# M<sup>2</sup>SE-VTTS: Multi-Modal and Multi-Scale Spatial Environment Understanding for Immersive Visual Text-to-Speech
+
+[![Paper](https://img.shields.io/badge/AAAI%202025-Paper-blue)](https://arxiv.org/abs/2412.11409)
+[![GitHub](https://img.shields.io/badge/GitHub-Code-green)](https://github.com/he-shuwei/M2SE-VTTS)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Visual Text-to-Speech (VTTS) aims to take the environmental image as the prompt to synthesize the reverberant speech for the spoken content. The challenge of this task lies in understanding the spatial environment from the image. Many attempts have been made to extract global spatial visual information from the RGB space of a spatial image. However, local and depth image information are crucial for understanding the spatial environment, which previous works have ignored. To address the issues, we propose a novel multi-modal and multi-scale spatial environment understanding scheme to achieve immersive VTTS, termed M<sup>2</sup>SE-VTTS. The multi-modal aims to take both the RGB and Depth spaces of the spatial image to learn more comprehensive spatial information, and the multi-scale seeks to model the local and global spatial knowledge simultaneously. Specifically, we first split the RGB and Depth images into patches and adopt the Gemini-generated environment captions to guide the local spatial understanding. After that, the multi-modal and multi-scale features are integrated by the local-aware global spatial understanding. In this way, M<sup>2</sup>SE-VTTS effectively models the interactions between local and global spatial contexts in the multi-modal spatial environment. Objective and subjective evaluations suggest that our model outperforms the advanced baselines in environmental speech generation.
+
+## Repository Contents
+
+| Resource | Path | Description |
+|---|---|---|
+| M<sup>2</sup>SE-VTTS | `m2se_vtts/` | Finetuned model checkpoint |
+| BigVGAN v2 | `bigvgan/` | Retrained 16 kHz vocoder |
+| Spatial environment captions | `data/raw_data/captions/` | Gemini-generated captions for all splits |
+| MFA alignment results | `data/processed_data/mfa/mfa_outputs.tar.gz` | Pre-computed forced alignment (TextGrid) |
+
+## Usage
+
+Please refer to the [GitHub repository](https://github.com/he-shuwei/M2SE-VTTS) for installation, training, and inference instructions.
+
+```bash
+git clone https://github.com/he-shuwei/M2SE-VTTS.git
+cd M2SE-VTTS
+
+# Download checkpoints
+# Place m2se_vtts/ and bigvgan/ under checkpoints/
+# Place captions under data/raw_data/captions/
+# Extract mfa_outputs.tar.gz to data/processed_data/mfa/outputs/
+
+# Inference
+bash scripts/infer/run_infer.sh \
+    --ckpt checkpoints/m2se_vtts/model_ckpt_best.pt \
+    --outdir results/m2se_vtts/test_seen \
+    --batch_size 16
+```
+
+## Citation
+
+```bibtex
+@inproceedings{liu2025multi,
+  title={Multi-modal and multi-scale spatial environment understanding for immersive visual text-to-speech},
+  author={Liu, Rui and He, Shuwei and Hu, Yifan and Li, Haizhou},
+  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
+  volume={39},
+  number={23},
+  pages={24632--24640},
+  year={2025}
+}
+```
+
+## Acknowledgments
+
+This project builds upon several excellent open-source projects:
+
+- [NATSpeech](https://github.com/NATSpeech/NATSpeech) — non-autoregressive TTS framework
+- [DiffSinger](https://github.com/MoonInTheRiver/DiffSinger) — diffusion-based acoustic model
+- [F5-TTS](https://github.com/SWivid/F5-TTS) — Diffusion Transformer (DiT) architecture
+- [BigVGAN](https://github.com/NVIDIA/BigVGAN) — neural vocoder by NVIDIA
+- [SoundSpaces-Speech](https://github.com/facebookresearch/learning-audio-visual-dereverberation) — audio-visual dataset by Meta Research
+- [CLIP](https://github.com/openai/CLIP) — visual-language encoder by OpenAI
+- [RMVPE](https://github.com/Dream-High/RMVPE) — robust pitch extractor