AudioLDM-ControlNet
Official model checkpoint for the paper:
Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment
Model Description
AudioLDM-ControlNet is a dance-conditioned music generation model. It combines:
- pretrained motion and music encoders,
- beat-guided contrastive pretraining for cross-modal alignment, and
- a ControlNet-style conditioning module built on an AudioLDM backbone.
The released checkpoint corresponds to the full model reported in the paper.
Repository Contents
The following files are stored in the root of this model repository:
| File | Description |
|---|
checkpoint-global_step=299999.ckpt | Full AudioLDM-ControlNet model checkpoint at training step 299999 |
audioldm_original_medium_stretch_pretrained.yaml | Model and preprocessing configuration corresponding to the checkpoint |
Checkpoint Details
- Framework: PyTorch / PyTorch Lightning
- Checkpoint format:
.ckpt
- Paper variant: Full model
- Training step: 299999
- Audio sampling rate: 16 kHz
- Motion input frame rate: 60 FPS
- Motion sequence length used by the paper configuration: 128 frames
The checkpoint contains a PyTorch Lightning-compatible state_dict.
Input Representation
The model is conditioned on human pose sequences represented as NumPy arrays with the following shape:
The 17 joints follow COCO joint order. The final dimension contains the per-joint coordinates expected by the official preprocessing and evaluation code.
Training Procedure
Training is performed in two stages:
- MotionBERT and MERT features are aligned using beat-guided contrastive pretraining based on BeatDance.
- The pretrained dance encoder conditions an AudioLDM backbone through a ControlNet adapter.
The implementation, preprocessing code, training scripts, and evaluation scripts are available in the official GitHub repository.
Citation
1@article{kimura2026dance,
2 title={Dance to Music Generation leveraging Pre-training with Unpaired Data and Contrastive Alignment},
3 author={Kimura, Ryota and Park, Sangheon and Polouliakh, Natalia and Akama, Taketo},
4 journal={arXiv preprint arXiv:2607.10537},
5 year={2026}
6}
Acknowledgements
This project builds on AudioLDM, ControlNet, MotionBERT, MERT, and BeatDance. Please cite the corresponding projects in addition to this work when using the model.
License
The source code repository is distributed under the MIT License. The model checkpoint may also be subject to the licenses and terms of the upstream pretrained models and datasets. Review the applicable upstream licenses before redistribution or commercial use.
