README.md

---
license: other
license_name: mixed-mit-cc-by-sa-3.0
library_name: motius
tags:
  - motion-generation
  - speech-to-gesture
  - diffusion
  - smpl-x
  - beat2
---

# DiffuseStyleGesture+ Model Card

DiffuseStyleGesture is the IJCAI 2023 work *Stylized Audio-Driven Co-Speech
Gesture Generation with Diffusion Models*. Motius integrates the released
DiffuseStyleGesture+ BEAT checkpoint, which adds text semantics, speaker ID,
30 seed frames, and rolling 150-frame diffusion windows.

- Paper: https://arxiv.org/abs/2305.04919
- DSG+ paper: https://arxiv.org/abs/2308.13879
- Official source: https://github.com/YoungSeng/DiffuseStyleGesture
- Motius release: https://huggingface.co/ZeyuLing/Motius-DiffuseStyleGesture-BEAT
- Pipeline: `motius.pipelines.diffusestylegesture.DiffuseStyleGesturePipeline`

## Release Snapshot

| Item | Value |
| ---- | ----- |
| Task | Speech-to-Gesture |
| Training data | BEAT speakers 2 and 10 |
| Benchmark | BEAT2 English speaker 2 |
| Input | WavLM/acoustic 1133D + aligned FastText 301D + speaker ID |
| Native motion | BEAT 75-joint BVH |
| Motius motion | SMPL-X 55-joint axis-angle + zero expression + translation |
| Motion rate | 30 FPS |
| Window | 150 frames, 30 seed frames, 120-frame stride |
| Diffusion | Official 1,000-step sampler, seed `123456` |
| Checkpoint | `model001080000.pt` |
| Checkpoint SHA-256 | `2e25d8b257b44e5110a61713476a146e99c34926c763b5dd1c18a6884a857c77` |
| Runtime revision | `9649995cdd0b8ce0d42bd02567b2bd03829a0540` |
| Inference-pinned artifact revision | `8707378e3835ac2d7e17dc977272640190a1daf5` |
| License | MIT model/runtime; CC-BY-SA-3.0 FastText subset |

## Setup

```bash
bash tools/setup_diffusestylegesture_env.sh
git clone https://github.com/YoungSeng/DiffuseStyleGesture \
  ref_repo/DiffuseStyleGesture
git -C ref_repo/DiffuseStyleGesture checkout \
  9649995cdd0b8ce0d42bd02567b2bd03829a0540
```

The exact original-format WavLM checkpoint is required. The Transformers
conversion is not used: on the released DSG+ parity clip its WavLM block has
only `0.96376` cosine similarity, while the original checkpoint reproduces the
released features at `0.9999999999` cosine similarity.

```bash
gdown 12-cB34qCTvByWT-QtOcZaqwwO21FLSqU \
  -O outputs/checkpoints/diffusestylegesture/WavLM-Large.pt
```

The FastText source is the official `crawl-300d-2M.vec.zip`. The release
contains the 790 vectors used by the fixed benchmark population as
`beat2_speaker2_fasttext_vocab.npz` (one source token, `berties`, is OOV).
The DSG+ checkpoint and source use MIT terms; this compact vector subset
retains FastText's CC-BY-SA-3.0 terms.

## Usage

Build the official conditioning features, then run inference:

```bash
python tools/build_diffusestylegesture_beat2_features.py \
  --runtime-root ref_repo/DiffuseStyleGesture/BEAT-TWH-main \
  --manifest outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/manifest.json \
  --data-root outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/beat_english_v2.0.0 \
  --wavlm-checkpoint outputs/checkpoints/diffusestylegesture/WavLM-Large.pt \
  --fasttext-compact outputs/checkpoints/diffusestylegesture/beat2_speaker2_fasttext_vocab.npz \
  --out-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/features

python tools/eval_diffusestylegesture_beat2.py \
  --runtime-root ref_repo/DiffuseStyleGesture/BEAT-TWH-main \
  --artifact ZeyuLing/Motius-DiffuseStyleGesture-BEAT \
  --manifest outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/manifest.json \
  --data-root outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/beat_english_v2.0.0 \
  --feature-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/features \
  --out-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/predictions
```

DSG+ writes a native 30 FPS, Y-up BEAT BVH. Motius applies the public
FreeTalker BEAT-to-SMPL-X joint map with identity axes and without FreeTalker's
historical six-frame downsample, which is only appropriate for original
120 FPS BEAT files.

Conditioning stays on the official audio-feature clock. The 15 benchmark WAVs
produce 6–16 more feature frames than their paired SMPL-X references; Motius
does not interpolate or time-warp those features. It runs diffusion for the
native feature length and crops only the final motion to the reference length.

## Evaluation Results

| Evaluator | Clips | FGD ↓ | BC ↑ | Diversity | uTMR FID ↓ | uTMR Paired Dist. ↓ |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| BEAT2 Official | 15 | 65.9364 | 6.3858 | 12.2055 | 0.0930 | 16.0251 |

The uTMR columns consume only canonical 30 FPS SMPL-22 joint positions. They
ignore text and expression coefficients; the official metric FK explicitly
sets the SMPL-X translation vector to zero.
Each long clip is split into balanced windows no longer than 20 seconds;
embeddings are L2-normalized only for FID. Paired distance and embedding
diversity use native uTMR features.

These values do not reproduce the DSG+ paper's FGD `14.461`: that number uses
the GENEA/Talking-With-Hands challenge autoencoder and test population, whereas
this row uses the fixed Motius BEAT2 speaker-2 SMPL-X protocol. The upstream
authors also document that the manual BVH-to-SMPL-X conversion can depress
motion realism on BEAT2, even for reconstruction. Accordingly, this released
row is a verified cross-protocol Motius result rather than a paper-number claim.
The exact GENEA autoencoder checkpoint and Table 1 split were not released, so
the public artifacts are insufficient for an exact `14.461` reproduction.

## Preview

Open the audio-synchronized
[15-case Three.js comparison](https://zeyuling-speech-to-gesture-beat2-leaderboard.static.hf.space/cases/index.html).
It provides shared camera and timeline controls for BEAT2 GT, Language of
Motion, EMAGE, and DiffuseStyleGesture+, plus native SMPL-X NPZ downloads.

## Citation

```bibtex
@inproceedings{yang2023diffusestylegesture,
  title={DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models},
  author={Yang, Sicheng and Wu, Zhiyong and Li, Minglei and Zhang, Zhensong and Hao, Lei and Bao, Weihong and Cheng, Ming and Xiao, Long},
  booktitle={IJCAI},
  year={2023}
}
```