| --- |
| license: mit |
| tags: |
| - dance |
| - pose-generation |
| - music |
| - transformer |
| - audio-to-motion |
| library_name: pytorch |
| pipeline_tag: other |
| --- |
| |
| # Dance2Music — Music2Pose Transformer |
|
|
| Generate dancing skeleton animations from any audio file using a causal transformer trained on [AIST++](https://google.github.io/aistplusplus_dataset/) dance videos. |
|
|
| **Author:** [Emma Genthon](https://huggingface.co/Emma5099) · genthonemma@gmail.com |
| **Best checkpoint:** epoch 180 (`pytorch_model.pt`) |
|
|
| ## What it does |
|
|
| Given a music track, the model autoregressively predicts 33 MediaPipe-style body landmarks at 25 fps, then renders them as a skeleton video synced to the audio. |
|
|
| ``` |
| Audio (32 kHz) → mel + onset + beat features → Music2PoseTransformer → poses (33×3) → skeleton video |
| ``` |
|
|
| ## Quick start |
|
|
| ```bash |
| git clone https://huggingface.co/Emma5099/dance2music |
| cd dance2music |
| pip install -r requirements.txt |
| |
| # Requires ffmpeg on PATH |
| python generate.py --audio your_song.wav --out dance.mp4 |
| ``` |
|
|
| ### Python API |
|
|
| ```python |
| import soundfile as sf |
| import librosa |
| import torch |
| |
| from audio_features import AUDIO_SR, audio_to_features |
| from inference import load_checkpoint, generate_poses |
| |
| waveform, sr = sf.read("song.wav", dtype="float32") |
| if waveform.ndim > 1: |
| waveform = waveform.mean(axis=1) |
| if sr != AUDIO_SR: |
| waveform = librosa.resample(waveform, orig_sr=sr, target_sr=AUDIO_SR) |
| |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
| model, ckpt = load_checkpoint("pytorch_model.pt", device) |
| poses = generate_poses( |
| model, audio_to_features(waveform), |
| ckpt["x_mean"], ckpt["x_std"], |
| ckpt["y_mean"], ckpt["y_std"], |
| device, |
| ) |
| # poses.shape → (T, 33, 3) |
| ``` |
|
|
| ## Model |
|
|
| | Property | Value | |
| |---|---| |
| | Architecture | Causal transformer decoder with audio cross-attention | |
| | Hidden size | 256 | |
| | Layers / heads | 6 / 8 | |
| | Audio input | 80 mel bands + onset strength + beat flag (82 dims/frame) | |
| | Pose output | 33 landmarks × 9 channels (xyz + velocity + acceleration), inference uses xyz only | |
| | Context window | 60 frames (2.4 s rolling buffer) | |
| | Training epochs | 180 (selected as best overall checkpoint) | |
|
|
| ### Validation metrics (sample `sBT_c01_v7`) |
|
|
| | Metric | Value | |
| |---|---| |
| | MSE (xyz) | 0.624 | |
| | Motion speed | 18.4% of ground truth | |
| | Balanced score | 1.306 | |
|
|
| The model produces human-shaped skeletons with visible motion. Beat alignment is still limited and improves with more training data and epochs. |
|
|
| ## Files |
|
|
| | File | Description | |
| |---|---| |
| | `pytorch_model.pt` | Best checkpoint (epoch 180) with weights + normalisation stats | |
| | `model.py` | `Music2PoseTransformer` architecture | |
| | `audio_features.py` | Mel/onset/beat feature extraction | |
| | `inference.py` | Checkpoint loading and autoregressive generation | |
| | `generate.py` | End-to-end CLI: audio → skeleton MP4 | |
| | `config.json` | Model hyperparameters and metadata | |
|
|
| ## Training (optional) |
|
|
| Training uses AIST++ pose sequences paired with audio chunks, weighted losses on position/velocity/acceleration, and scheduled sampling with a rolling 60-frame context. See the full project source for `train.py`, `evaluate.py`, and dataset tooling. |
|
|
| ## Requirements |
|
|
| - Python 3.10+ |
| - PyTorch 2.x |
| - ffmpeg (for video rendering and audio muxing) |
| - ~26 MB disk for model weights |
|
|
| ## License |
|
|
| MIT |
|
|
| ## Citation |
|
|
| If you use this model, please cite AIST++ and this repository: |
|
|
| ```bibtex |
| @misc{genthon2026dance2music, |
| author = {Emma Genthon}, |
| title = {Dance2Music: Music-Conditioned Pose Generation}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| howpublished = {\url{https://huggingface.co/Emma5099/dance2music}} |
| } |
| ``` |
|
|
