--- license: other license_name: mixed-mit-cc-by-sa-3.0 library_name: motius tags: - motion-generation - speech-to-gesture - diffusion - smpl-x - beat2 --- # DiffuseStyleGesture+ Model Card DiffuseStyleGesture is the IJCAI 2023 work *Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models*. Motius integrates the released DiffuseStyleGesture+ BEAT checkpoint, which adds text semantics, speaker ID, 30 seed frames, and rolling 150-frame diffusion windows. - Paper: https://arxiv.org/abs/2305.04919 - DSG+ paper: https://arxiv.org/abs/2308.13879 - Official source: https://github.com/YoungSeng/DiffuseStyleGesture - Motius release: https://huggingface.co/ZeyuLing/Motius-DiffuseStyleGesture-BEAT - Pipeline: `motius.pipelines.diffusestylegesture.DiffuseStyleGesturePipeline` ## Release Snapshot | Item | Value | | ---- | ----- | | Task | Speech-to-Gesture | | Training data | BEAT speakers 2 and 10 | | Benchmark | BEAT2 English speaker 2 | | Input | WavLM/acoustic 1133D + aligned FastText 301D + speaker ID | | Native motion | BEAT 75-joint BVH | | Motius motion | SMPL-X 55-joint axis-angle + zero expression + translation | | Motion rate | 30 FPS | | Window | 150 frames, 30 seed frames, 120-frame stride | | Diffusion | Official 1,000-step sampler, seed `123456` | | Checkpoint | `model001080000.pt` | | Checkpoint SHA-256 | `2e25d8b257b44e5110a61713476a146e99c34926c763b5dd1c18a6884a857c77` | | Runtime revision | `9649995cdd0b8ce0d42bd02567b2bd03829a0540` | | Inference-pinned artifact revision | `8707378e3835ac2d7e17dc977272640190a1daf5` | | License | MIT model/runtime; CC-BY-SA-3.0 FastText subset | ## Setup ```bash bash tools/setup_diffusestylegesture_env.sh git clone https://github.com/YoungSeng/DiffuseStyleGesture \ ref_repo/DiffuseStyleGesture git -C ref_repo/DiffuseStyleGesture checkout \ 9649995cdd0b8ce0d42bd02567b2bd03829a0540 ``` The exact original-format WavLM checkpoint is required. The Transformers conversion is not used: on the released DSG+ parity clip its WavLM block has only `0.96376` cosine similarity, while the original checkpoint reproduces the released features at `0.9999999999` cosine similarity. ```bash gdown 12-cB34qCTvByWT-QtOcZaqwwO21FLSqU \ -O outputs/checkpoints/diffusestylegesture/WavLM-Large.pt ``` The FastText source is the official `crawl-300d-2M.vec.zip`. The release contains the 790 vectors used by the fixed benchmark population as `beat2_speaker2_fasttext_vocab.npz` (one source token, `berties`, is OOV). The DSG+ checkpoint and source use MIT terms; this compact vector subset retains FastText's CC-BY-SA-3.0 terms. ## Usage Build the official conditioning features, then run inference: ```bash python tools/build_diffusestylegesture_beat2_features.py \ --runtime-root ref_repo/DiffuseStyleGesture/BEAT-TWH-main \ --manifest outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/manifest.json \ --data-root outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/beat_english_v2.0.0 \ --wavlm-checkpoint outputs/checkpoints/diffusestylegesture/WavLM-Large.pt \ --fasttext-compact outputs/checkpoints/diffusestylegesture/beat2_speaker2_fasttext_vocab.npz \ --out-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/features python tools/eval_diffusestylegesture_beat2.py \ --runtime-root ref_repo/DiffuseStyleGesture/BEAT-TWH-main \ --artifact ZeyuLing/Motius-DiffuseStyleGesture-BEAT \ --manifest outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/manifest.json \ --data-root outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/beat_english_v2.0.0 \ --feature-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/features \ --out-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/predictions ``` DSG+ writes a native 30 FPS, Y-up BEAT BVH. Motius applies the public FreeTalker BEAT-to-SMPL-X joint map with identity axes and without FreeTalker's historical six-frame downsample, which is only appropriate for original 120 FPS BEAT files. Conditioning stays on the official audio-feature clock. The 15 benchmark WAVs produce 6–16 more feature frames than their paired SMPL-X references; Motius does not interpolate or time-warp those features. It runs diffusion for the native feature length and crops only the final motion to the reference length. ## Evaluation Results | Evaluator | Clips | FGD ↓ | BC ↑ | Diversity | uTMR FID ↓ | uTMR Paired Dist. ↓ | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | BEAT2 Official | 15 | 65.9364 | 6.3858 | 12.2055 | 0.0930 | 16.0251 | The uTMR columns consume only canonical 30 FPS SMPL-22 joint positions. They ignore text and expression coefficients; the official metric FK explicitly sets the SMPL-X translation vector to zero. Each long clip is split into balanced windows no longer than 20 seconds; embeddings are L2-normalized only for FID. Paired distance and embedding diversity use native uTMR features. These values do not reproduce the DSG+ paper's FGD `14.461`: that number uses the GENEA/Talking-With-Hands challenge autoencoder and test population, whereas this row uses the fixed Motius BEAT2 speaker-2 SMPL-X protocol. The upstream authors also document that the manual BVH-to-SMPL-X conversion can depress motion realism on BEAT2, even for reconstruction. Accordingly, this released row is a verified cross-protocol Motius result rather than a paper-number claim. The exact GENEA autoencoder checkpoint and Table 1 split were not released, so the public artifacts are insufficient for an exact `14.461` reproduction. ## Preview Open the audio-synchronized [15-case Three.js comparison](https://zeyuling-speech-to-gesture-beat2-leaderboard.static.hf.space/cases/index.html). It provides shared camera and timeline controls for BEAT2 GT, Language of Motion, EMAGE, and DiffuseStyleGesture+, plus native SMPL-X NPZ downloads. ## Citation ```bibtex @inproceedings{yang2023diffusestylegesture, title={DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models}, author={Yang, Sicheng and Wu, Zhiyong and Li, Minglei and Zhang, Zhensong and Hao, Lei and Bao, Weihong and Cheng, Ming and Xiao, Long}, booktitle={IJCAI}, year={2023} } ```