AudioLDM-ControlNet

Official model checkpoint for the paper:

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

• Paper: https://huggingface.co/papers/2607.10537
• arXiv: https://arxiv.org/abs/2607.10537
• Source code: https://github.com/kmraven/AudioLDM-ControlNet

Model Description

AudioLDM-ControlNet is a dance-conditioned music generation model. It combines:

1. pretrained motion and music encoders,
2. beat-guided contrastive pretraining for cross-modal alignment, and
3. a ControlNet-style conditioning module built on an AudioLDM backbone.

The released checkpoint corresponds to the full model reported in the paper.

Repository Contents

The following files are stored in the root of this model repository:

File	Description
checkpoint-global_step=299999.ckpt	Full AudioLDM-ControlNet model checkpoint at training step 299999
audioldm_original_medium_stretch_pretrained.yaml	Model and preprocessing configuration corresponding to the checkpoint

Checkpoint Details

• Framework: PyTorch / PyTorch Lightning
• Checkpoint format: .ckpt
• Paper variant: Full model
• Training step: 299999
• Audio sampling rate: 16 kHz
• Motion input frame rate: 60 FPS
• Motion sequence length used by the paper configuration: 128 frames

The checkpoint contains a PyTorch Lightning-compatible state_dict.

Input Representation

The model is conditioned on human pose sequences represented as NumPy arrays with the following shape:

[frames, 17, 3]

The 17 joints follow COCO joint order. The final dimension contains the per-joint coordinates expected by the official preprocessing and evaluation code.

Training Procedure

Training is performed in two stages:

1. MotionBERT and MERT features are aligned using beat-guided contrastive pretraining based on BeatDance.
2. The pretrained dance encoder conditions an AudioLDM backbone through a ControlNet adapter.

The implementation, preprocessing code, training scripts, and evaluation scripts are available in the official GitHub repository.

Citation

@article{kimura2026dance,
  title={Dance to Music Generation leveraging Pre-training with Unpaired Data and Contrastive Alignment},
  author={Kimura, Ryota and Park, Sangheon and Polouliakh, Natalia and Akama, Taketo},
  journal={arXiv preprint arXiv:2607.10537},
  year={2026}
}

Acknowledgements

This project builds on AudioLDM, ControlNet, MotionBERT, MERT, and BeatDance. Please cite the corresponding projects in addition to this work when using the model.

License

The source code repository is distributed under the MIT License. The model checkpoint may also be subject to the licenses and terms of the upstream pretrained models and datasets. Review the applicable upstream licenses before redistribution or commercial use.