README.md

---
title: GestureLSM
emoji: 🤸
colorFrom: yellow
colorTo: green
sdk: gradio
sdk_version: 5.23.3
app_file: app.py
python_version: "3.10"
short_description: Latent Shortcut based Co-Speech Gesture Generation
startup_duration_timeout: 30m
---

# GestureLSM: Latent Shortcut based Co-Speech Gesture Generation

This Space demonstrates **GestureLSM**, a flow-matching-based framework for co-speech gesture generation. Given an audio clip, the model generates realistic full-body gestures synchronized with the speech.

## How it works

1. Upload an audio clip (or use one of the provided examples)
2. The model extracts audio features (onset + amplitude) and word-level text features (via Whisper ASR)
3. A spatial-temporal flow-matching model generates gesture latents
4. Pretrained RVQVAE decoders convert latents to body-part-specific poses (upper body, hands, lower body)
5. SMPLX model renders the poses to a 3D mesh, which is rendered to video

## Notes

- Face animation is fixed; only body motion is generated
- Inference takes 1-4 minutes depending on audio length
- The model uses the BEAT dataset's speaker-independent weights from [pliu23/GestureLSM](https://huggingface.co/pliu23/GestureLSM)

## References

- Paper: [GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling](https://arxiv.org/abs/2501.18898)
- Code: [GitHub](https://github.com/andypinxinliu/GestureLSM)