AudioLDM-ControlNet

Official checkpoints and evaluation artifacts for the paper:

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

• Paper: https://huggingface.co/papers/2607.10537
• arXiv: https://arxiv.org/abs/2607.10537
• Source code: https://github.com/kmraven/AudioLDM-ControlNet

Model Description

AudioLDM-ControlNet is a dance-conditioned music generation model. It combines:

1. pretrained motion and music encoders,
2. beat-guided contrastive pretraining for cross-modal alignment, and
3. a ControlNet-style conditioning module built on an AudioLDM backbone.

This repository includes the full model, the w/o MotionBERT checkpoint, along with generated audio, evaluation metrics, training logs, and comparison videos.

Repository Contents

Model checkpoints and evaluation outputs are organized under models/ and testsets_video_results/.

Path	Contents
models/full/	Full-model checkpoint, YAML configuration, 36 generated WAV files and metrics.txt in 300ksteps/, and the W&B training run in wandb/
models/without_motioinbert/	w/o MotionBERT checkpoint, YAML configuration, and 36 generated WAV files and metrics.txt in 300ksteps/
testsets_video_results/	216 MP4 comparison videos: 36 clips for each of the six conditions listed below
checkpoint-global_step=299999.ckpt	Full-model checkpoint, also available under models/full/checkpoints/
audioldm_original_medium_stretch_pretrained.yaml	Full-model configuration, also available under models/full/

The root-level checkpoint and configuration are identical to their counterparts under models/full/.

Both checkpoints are named checkpoint-global_step=299999.ckpt and are stored in their respective variant's checkpoints/ directory. The matching configurations are:

• Full model: models/full/audioldm_original_medium_stretch_pretrained.yaml
• w/o MotionBERT: models/without_motioinbert/audioldm_original_medium_stretch_wo_mb.yaml

The six video folders are AudioLDM (default), Ground Truth, Li et al., Ours, Ours (w-o Contrastive Pretraining), and Ours (w-o MotionBERT). The w/o Contrastive Pretraining variant is represented by comparison videos; a checkpoint for that variant is not included in this release.

Download and Use

Install the Hugging Face CLI if needed:

pip install -U huggingface_hub

Download all model and evaluation files (approximately 16.5 GB):

hf download kmraven/AudioLDM-ControlNet \
  --include "models/*" --include "testsets_video_results/*" \
  --local-dir data/releases/AudioLDM-ControlNet

To download only the full-model checkpoint and configuration:

hf download kmraven/AudioLDM-ControlNet \
  "models/full/checkpoints/checkpoint-global_step=299999.ckpt" \
  "models/full/audioldm_original_medium_stretch_pretrained.yaml" \
  --local-dir data/releases/AudioLDM-ControlNet

Run preprocessing and evaluation with the official source code and setup instructions. These are project-specific PyTorch Lightning checkpoints; use the repository's evaluation scripts. The YAML files specify the training settings and use relative paths. Dataset inputs and separately referenced upstream weights must be prepared as described in the source repository. This release does not include a standalone Stage 1 model_best.pth checkpoint.

Exact training resume, including optimizer state and global step, may be incompatible with the current training code. If exact resume fails, the source repository recommends commit 0c1f4e3197a7c339586ce5769c6a4510dfbc6f30.

Checkpoint Details

• Framework: PyTorch / PyTorch Lightning
• Checkpoint format: .ckpt
• Paper variants with checkpoints: Full model and w/o MotionBERT
• Training step: 299999
• Audio sampling rate: 16 kHz
• Motion input frame rate: 60 FPS
• Motion sequence length used by the paper configuration: 128 frames

The checkpoint contains a PyTorch Lightning-compatible state_dict.

Input Representation

The model is conditioned on human pose sequences represented as NumPy arrays with the following shape:

[frames, 17, 3]

The 17 joints follow COCO joint order. The final dimension contains the per-joint coordinates expected by the official preprocessing and evaluation code.

Training Procedure

Training is performed in two stages:

1. MotionBERT and MERT features are aligned using beat-guided contrastive pretraining based on BeatDance.
2. The pretrained dance encoder conditions an AudioLDM backbone through a ControlNet adapter.

The implementation, preprocessing code, training scripts, and evaluation scripts are available in the official GitHub repository.

Citation

@article{kimura2026dance,
  title={Dance to Music Generation leveraging Pre-training with Unpaired Data and Contrastive Alignment},
  author={Kimura, Ryota and Park, Sangheon and Polouliakh, Natalia and Akama, Taketo},
  journal={arXiv preprint arXiv:2607.10537},
  year={2026}
}

Acknowledgements

This project builds on AudioLDM, ControlNet, MotionBERT, MERT, and BeatDance. Please cite the corresponding projects in addition to this work when using the model.

License

The source code repository is distributed under the MIT License. The model checkpoint may also be subject to the licenses and terms of the upstream pretrained models and datasets. Review the applicable upstream licenses before redistribution or commercial use.