Request access to the AMUSE research artifacts

Access is manually reviewed. For questions, contact amuse@tue.mpg.de or chhatre@kth.se.

By requesting access you agree to use these artifacts only under the AMUSE non-commercial research license.

AMUSE LPDM

Paper page: Hugging Face Papers 2312.04466
ArXiv: 2312.04466

This gated model repo contains the released AMUSE LPDM checkpoint folder LPDM_20231028-210758_actors_smplx plus a lightweight NPZ-only inference wrapper.

What is included

saved-models/LPDM_20231028-210758_actors_smplx/
configs/, dm/, and models/ required for inference
infer_npz.py
configs/release_base.json

What is not included

SMPL-X model files
Blender resources and Blender add-ons
Purchased third-party assets

SMPL-X and Blender are not needed to generate motion NPZs. The released inference path outputs:

poses: axis-angle joint rotations with shape [T, 55, 3]
trans: root translation with shape [T, 3]
mocap_frame_rate: 30.0

Quick start

python infer_npz.py \
  --audio /path/to/input.wav \
  --output /path/to/output_motion.npz

The script uses the local LPDM checkpoint in this repo and will fetch the gated audio checkpoint from kiranchhatre/amuse-audio automatically if it is not already available locally.

Notes

The released model is tuned for 10-second chunks at 16 kHz audio.
Longer audio is processed chunk-by-chunk and concatenated in time.
No Blender or SMPL-X downloads are required for this NPZ-only path.

Citation

If you use these artifacts, please cite AMUSE:

@InProceedings{Chhatre_2024_CVPR,
    author    = {Chhatre, Kiran and Daněček, Radek and Athanasiou, Nikos and Becherini, Giorgio and Peters, Christopher and Black, Michael J. and Bolkart, Timo},
    title     = {{AMUSE}: Emotional Speech-driven {3D} Body Animation via Disentangled Latent Diffusion},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {1942-1953},
    url       = {https://amuse.is.tue.mpg.de},
}

Downloads last month: 2

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train kiranchhatre/amuse-lpdm

Paper for kiranchhatre/amuse-lpdm

Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion

Paper • 2312.04466 • Published Dec 7, 2023