AudioLDM ControlNet
Official checkpoints and evaluation artifacts for the paper:
Model source
- Kind
- Other
- File size
- 7.48 GB
- Version
- vRevision 81505b7
- Updated
- Oct 9, 2026
- Downloads
- 0
- Source
- Hugging Face
Source description
AudioLDM-ControlNet
Official checkpoints and evaluation artifacts for the paper:
Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment
- Paper: https://huggingface.co/papers/2607.10537
- arXiv: https://arxiv.org/abs/2607.10537
- Source code: https://github.com/kmraven/AudioLDM-ControlNet
Model Description
AudioLDM-ControlNet is a dance-conditioned music generation model. It combines:
- pretrained motion and music encoders,
- beat-guided contrastive pretraining for cross-modal alignment, and
- a ControlNet-style conditioning module built on an AudioLDM backbone.
This repository includes the full model, the w/o MotionBERT checkpoint, along with generated audio, evaluation metrics, training logs, and comparison videos.
Repository Contents
Model checkpoints and evaluation outputs are organized under models/ and
testsets_video_results/.
| Path | Contents |
|---|---|
models/full/ | Full-model checkpoint, YAML configuration, 36 generated WAV files and metrics.txt in 300ksteps/, and the W&B training run in wandb/ |
models/without_motioinbert/ | w/o MotionBERT checkpoint, YAML configuration, and 36 generated WAV files and metrics.txt in 300ksteps/ |
testsets_video_results/ | 216 MP4 comparison videos: 36 clips for each of the six conditions listed below |
checkpoint-global_step=299999.ckpt | Full-model checkpoint, also available under models/full/checkpoints/ |
audioldm_original_medium_stretch_pretrained.yaml | Full-model configuration, also available under models/full/ |
The root-level checkpoint and configuration are identical to their counterparts
under models/full/.
Both checkpoints are named checkpoint-global_step=299999.ckpt and are stored
in their respective variant's checkpoints/ directory. The matching
configurations are:
- Full model:
models/full/audioldm_original_medium_stretch_pretrained.yaml - w/o MotionBERT:
models/without_motioinbert/audioldm_original_medium_stretch_wo_mb.yaml
The six video folders are AudioLDM (default), Ground Truth, Li et al.,
Ours, Ours (w-o Contrastive Pretraining), and Ours (w-o MotionBERT).
The w/o Contrastive Pretraining variant is represented by comparison videos;
a checkpoint for that variant is not included in this release.
Download and Use
Install the Hugging Face CLI if needed:
pip install -U huggingface_hub
Download all model and evaluation files (approximately 16.5 GB):
hf download kmraven/AudioLDM-ControlNet \
--include "models/*" --include "testsets_video_results/*" \
--local-dir data/releases/AudioLDM-ControlNet
To download only the full-model checkpoint and configuration:
hf download kmraven/AudioLDM-ControlNet \
"models/full/checkpoints/checkpoint-global_step=299999.ckpt" \
"models/full/audioldm_original_medium_stretch_pretrained.yaml" \
--local-dir data/releases/AudioLDM-ControlNet
Run preprocessing and evaluation with the
official source code and setup instructions.
These are project-specific PyTorch Lightning checkpoints; use the repository's
evaluation scripts. The YAML files specify the training settings and use
relative paths. Dataset inputs and separately referenced upstream weights must
be prepared as described in the source repository. This release
does not include a standalone Stage 1 model_best.pth checkpoint.
Exact training resume, including optimizer state and global step, may be
incompatible with the current training code. If exact resume fails, the source
repository recommends commit 0c1f4e3197a7c339586ce5769c6a4510dfbc6f30.
Checkpoint Details
- Framework: PyTorch / PyTorch Lightning
- Checkpoint format:
.ckpt - Paper variants with checkpoints: Full model and w/o MotionBERT
- Training step: 299999
- Audio sampling rate: 16 kHz
- Motion input frame rate: 60 FPS
- Motion sequence length used by the paper configuration: 128 frames
The checkpoint contains a PyTorch Lightning-compatible state_dict.
Input Representation
The model is conditioned on human pose sequences represented as NumPy arrays with the following shape:
[frames, 17, 3]
The 17 joints follow COCO joint order. The final dimension contains the per-joint coordinates expected by the official preprocessing and evaluation code.
Training Procedure
Training is performed in two stages:
- MotionBERT and MERT features are aligned using beat-guided contrastive pretraining based on BeatDance.
- The pretrained dance encoder conditions an AudioLDM backbone through a ControlNet adapter.
The implementation, preprocessing code, training scripts, and evaluation scripts are available in the official GitHub repository.
Citation
@article{kimura2026dance,
title={Dance to Music Generation leveraging Pre-training with Unpaired Data and Contrastive Alignment},
author={Kimura, Ryota and Park, Sangheon and Polouliakh, Natalia and Akama, Taketo},
journal={arXiv preprint arXiv:2607.10537},
year={2026}
}
Acknowledgements
This project builds on AudioLDM, ControlNet, MotionBERT, MERT, and BeatDance. Please cite the corresponding projects in addition to this work when using the model.
License
The source code repository is distributed under the MIT License. The model checkpoint may also be subject to the licenses and terms of the upstream pretrained models and datasets. Review the applicable upstream licenses before redistribution or commercial use.
Sources
1 source
Sources
1 sourceVerified Oct 8
Model artifacts
3 artifacts
Model artifacts
3 artifactsSource excerpts
3 excerpts
Source excerpts
3 excerpts--- license: other library_name: pytorch tags: - audio-generation - music-generation - dance-to-music - motion-conditioned-generation - controlnet - audioldm - pytorch-lightning - arxiv:2607.10537 --- # AudioLDM-ControlNet Official checkpoints and evaluation artifacts for the paper: **Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment** - Paper: https://huggingface.co/papers/2607.10537 - arXiv: https://arxiv.org/abs/2607.10537 - Source code: https://github.com/kmraven/AudioLDM-ControlNet ## Model Description AudioLDM-ControlNet is a dance-conditioned music generation model. It combines: 1. pretrained motion and music encoders, 2. beat-guided contrastive pretraining for cross-modal alignment, and 3. a ControlNet-style conditioning module built on an AudioLDM backbone. This repository includes the **full model**, the **w/o MotionBERT** checkpoint, along with generated audio, evaluation metrics, training logs, and comparison videos. ## Repository Contents Model checkpoints and evaluation outputs are organized under `models/` and `testsets_video_results/`. | Path | Contents | | --- | --- | | [`models/full/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/models/full) | Full-model checkpoint, YAML configuration, 36 generated WAV files and `metrics.txt` in `300ksteps/`, and the W&B training run in `wandb/` | | [`models/without_motioinbert/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/models/without_motioinbert) | w/o MotionBERT checkpoint, YAML configuration, and 36 generated WAV files and `metrics.txt` in `300ksteps/` | | [`testsets_video_results/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/testsets_video_results) | 216 MP4 comparison videos: 36 clips for each of the six conditions listed below | | `checkpoint-global_step=299999.ckpt` | Full-model checkpoint, also available under `models/full/checkpoints/` | | `audioldm_original_medium_stretch_pretrained.yaml` | Full-model configuration, also available under `models/full/` | The root-level checkpoint and configuration are identical to their counterparts under `models/full/`. Both checkpoints are named `checkpoint-global_step=299999.ckpt` and are stored in their respective variant's `checkpoints/` directory. The matching configurations are: - Full model: `models/full/audioldm_original_medium_stre...
--- license: other library_name: pytorch tags: - audio-generation - music-generation - dance-to-music - motion-conditioned-generation - controlnet - audioldm - pytorch-lightning - arxiv:2607.10537 --- # AudioLDM-ControlNet Official model checkpoint for the paper: **Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment** - Paper: https://huggingface.co/papers/2607.10537 - arXiv: https://arxiv.org/abs/2607.10537 - Source code: https://github.com/kmraven/AudioLDM-ControlNet ## Model Description AudioLDM-ControlNet is a dance-conditioned music generation model. It combines: 1. pretrained motion and music encoders, 2. beat-guided contrastive pretraining for cross-modal alignment, and 3. a ControlNet-style conditioning module built on an AudioLDM backbone. The released checkpoint corresponds to the **full model** reported in the paper. ## Repository Contents The following files are stored in the root of this model repository: | File | Description | | --- | --- | | `checkpoint-global_step=299999.ckpt` | Full AudioLDM-ControlNet model checkpoint at training step 299999 | | `audioldm_original_medium_stretch_pretrained.yaml` | Model and preprocessing configuration corresponding to the checkpoint | ## Checkpoint Details - Framework: PyTorch / PyTorch Lightning - Checkpoint format: `.ckpt` - Paper variant: Full model - Training step: 299999 - Audio sampling rate: 16 kHz - Motion input frame rate: 60 FPS - Motion sequence length used by the paper configuration: 128 frames The checkpoint contains a PyTorch Lightning-compatible `state_dict`. ## Input Representation The model is conditioned on human pose sequences represented as NumPy arrays with the following shape: ```text [frames, 17, 3] ``` The 17 joints follow COCO joint order. The final dimension contains the per-joint coordinates expected by the official preprocessing and evaluation code. ## Training Procedure Training is performed in two stages: 1. MotionBERT and MERT features are aligned using beat-guided contrastive pretraining based on BeatDance. 2. The pretrained dance encoder conditions an AudioLDM backbone through a ControlNet adapter. The implementation, preprocessing code, training scripts, and evaluation scripts are available in the official GitHub repository. ## Citation ```bibtex @article{kimura2026dance, title={Dance to Music Ge...
Source context: 0 downloads · 0 likes · Library pytorch · Repo kmraven/AudioLDM-ControlNet
