README.md

metadata

title: GestureLSM
emoji: 🤸
colorFrom: yellow
colorTo: green
sdk: gradio
sdk_version: 5.23.3
app_file: app.py
python_version: '3.10'
short_description: Latent Shortcut based Co-Speech Gesture Generation
startup_duration_timeout: 30m

GestureLSM: Latent Shortcut based Co-Speech Gesture Generation

This Space demonstrates GestureLSM, a flow-matching-based framework for co-speech gesture generation. Given an audio clip, the model generates realistic full-body gestures synchronized with the speech.

How it works

1. Upload an audio clip (or use one of the provided examples)
2. The model extracts audio features (onset + amplitude) and word-level text features (via Whisper ASR)
3. A spatial-temporal flow-matching model generates gesture latents
4. Pretrained RVQVAE decoders convert latents to body-part-specific poses (upper body, hands, lower body)
5. SMPLX model renders the poses to a 3D mesh, which is rendered to video

Notes

• Face animation is fixed; only body motion is generated
• Inference takes 1-4 minutes depending on audio length
• The model uses the BEAT dataset's speaker-independent weights from pliu23/GestureLSM

References

• Paper: GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling
• Code: GitHub