Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion
Paper • 2312.04466 • Published
Request access to the AMUSE research artifacts
Access is manually reviewed. For questions, contact amuse@tue.mpg.de or chhatre@kth.se.
By requesting access you agree to use these artifacts only under the AMUSE non-commercial research license.
Log in or Sign Up to review the conditions and access this model content.
AMUSE Audio Encoder
Paper page: Hugging Face Papers 2312.04466
ArXiv: 2312.04466
This gated model repo contains the released AMUSE audio checkpoint folder wav_dtw_mfcc_20231022-044436_actors.
It is used together with kiranchhatre/amuse-lpdm for speech-to-motion inference. The released checkpoint
supports the AMUSE CVPR 2024 setup used by the LPDM release in LPDM_20231028-210758_actors_smplx.
What is included
saved-models/wav_dtw_mfcc_20231022-044436_actors/
What is not included
- SMPL-X assets
- Blender resources
- Auto-Rig Pro or other third-party add-ons
Those assets are not required for NPZ-only inference. They are only needed for downstream mesh or Blender rendering workflows.
Usage
Use the infer_npz.py wrapper from kiranchhatre/amuse-lpdm. It will download this audio repo
automatically after you have authenticated with Hugging Face and been granted access.
Citation
If you use these artifacts, please cite AMUSE:
@InProceedings{Chhatre_2024_CVPR,
author = {Chhatre, Kiran and Daněček, Radek and Athanasiou, Nikos and Becherini, Giorgio and Peters, Christopher and Black, Michael J. and Bolkart, Timo},
title = {{AMUSE}: Emotional Speech-driven {3D} Body Animation via Disentangled Latent Diffusion},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
pages = {1942-1953},
url = {https://amuse.is.tue.mpg.de},
}
- Downloads last month
- 2
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support