README.md

metadata

extra_gated_eu_disallowed: true

Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models

Haoyu Zhang¹, Zhipeng Li², Yiwen Guo^3†, Tianshu Yu^1†
1The Chinese University of Hong Kong, Shenzhen    ²LIGHTSPEED    ³Independent Researcher
^†Corresponding Authors

---

Ex-Omni is an public release for omni-modal response generation. Given text or speech input, the system can produce response text, speech units / decoded audio, and 52-dimensional facial blendshape coefficients, with optional rendering into a talking-face video.

📖 Table of Contents

• Repository Structure
• Quick Start
  • Installation
  • Checkpoints and Assets
  • Launch the Demo
• Acknowledgements
• Citation

🗂️ Repository Structure

.
├── asset/                              # Download the mesh templates here
├── ckpt/                               # Download the checkpoints here
├── cosyvoice/                          # Runtime audio decoder modules
├── deploy.py                           # Main Gradio entrypoint
├── deploy_base.py                      # Shared inference pipeline and UI logic
├── ex_omni/
│   ├── constants.py                    # Runtime constants
│   ├── flow_inference.py               # Audio decoder wrapper
│   ├── render_utils.py                 # Blendshape rendering utilities
│   └── model/
│       ├── language_model/             # Omni model wrapper
│       ├── speech_encoder/             # Whisper speech encoder
│       ├── speech_projector/           # Speech projector
│       ├── speech_generator/           # Speech generator
│       └── blendshape_generator/       # Blendshape generator
├── requirements.txt                    # Python dependencies
└── LICENSE.txt                         # License file

⚙️ Quick Start

Installation

# 1. Create and activate environment
conda create -n Ex-Omni python=3.10 -y
conda activate Ex-Omni

# 2. Install PyTorch (example: CUDA 12.6)
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu1216

# 3. Install project dependencies
pip install -r requirements.txt

# 4. Install pytorch3d separately according to your CUDA / PyTorch version
# See official pytorch3d installation instructions

Checkpoints and Assets

Prepare the following assets before running inference:

Component	Expected Path	Source
Ex-Omni checkpoint	ckpt/Ex-Omni/
Flow decoder checkpoint	ckpt/glm-4-voice-decoder/flow.pt
HiFT decoder checkpoint	ckpt/glm-4-voice-decoder/hift.pt
EmoTalk mesh template	asset/EmoTalk.npz
Claire mesh template	asset/claire.npz

For EmoTalk template, please download the resources.zip from UniTalker and extract the EmoTalk.npz file. For Claire template, please obtain the original Claire asset yourself and convert it into the .npz format.

Launch the Demo

python deploy.py \
  --model-path ckpt/Ex-Omni \
  --flow_ckpt_path ckpt/glm-4-voice-decoder/flow.pt \
  --hift_ckpt_path ckpt/glm-4-voice-decoder/hift.pt \
  --template_type emotalk \
  --port 8080

Then open:

http://localhost:8080

🙏 Acknowledgements

We would like to thank the authors of OpenOmni, LLaMA-Omni2, EmoTalk and UniTalker. Parts of the implementation and overall system design were developed with reference to their open-source release.

📄 Citation

If you use this project, please cite our paper:

@misc{zhang2026exomnienabling3dfacial,
      title={Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models}, 
      author={Haoyu Zhang and Zhipeng Li and Yiwen Guo and Tianshu Yu},
      year={2026},
      eprint={2602.07106},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2602.07106}, 
}