ModelPublicadoExterno

AudioLDM ControlNet

Official checkpoints and evaluation artifacts for the paper:

Fuentehuggingface.co/kmraven/AudioLDM-ControlNet/tree/81505b7066e04df833db794e856f51de996cdece

TipoOtherVersiónRevision 81505b7Ccalidad

Fuente del modelo

Tipo: Other
Tamaño del archivo: 7,48 GB
Versión: vRevision 81505b7
Actualizado: 9 oct 2026
Descargas: 0
Fuente: Hugging Face

Descripción de la fuente

AudioLDM-ControlNet

Official checkpoints and evaluation artifacts for the paper:

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

Paper: https://huggingface.co/papers/2607.10537
arXiv: https://arxiv.org/abs/2607.10537
Source code: https://github.com/kmraven/AudioLDM-ControlNet

Model Description

AudioLDM-ControlNet is a dance-conditioned music generation model. It combines:

pretrained motion and music encoders,
beat-guided contrastive pretraining for cross-modal alignment, and
a ControlNet-style conditioning module built on an AudioLDM backbone.

This repository includes the full model, the w/o MotionBERT checkpoint, along with generated audio, evaluation metrics, training logs, and comparison videos.

Repository Contents

Model checkpoints and evaluation outputs are organized under models/ and testsets_video_results/.

Path	Contents
`models/full/`	Full-model checkpoint, YAML configuration, 36 generated WAV files and `metrics.txt` in `300ksteps/`, and the W&B training run in `wandb/`
`models/without_motioinbert/`	w/o MotionBERT checkpoint, YAML configuration, and 36 generated WAV files and `metrics.txt` in `300ksteps/`
`testsets_video_results/`	216 MP4 comparison videos: 36 clips for each of the six conditions listed below
`checkpoint-global_step=299999.ckpt`	Full-model checkpoint, also available under `models/full/checkpoints/`
`audioldm_original_medium_stretch_pretrained.yaml`	Full-model configuration, also available under `models/full/`

The root-level checkpoint and configuration are identical to their counterparts under models/full/.

Both checkpoints are named checkpoint-global_step=299999.ckpt and are stored in their respective variant's checkpoints/ directory. The matching configurations are:

Full model: models/full/audioldm_original_medium_stretch_pretrained.yaml
w/o MotionBERT: models/without_motioinbert/audioldm_original_medium_stretch_wo_mb.yaml

The six video folders are AudioLDM (default), Ground Truth, Li et al., Ours, Ours (w-o Contrastive Pretraining), and Ours (w-o MotionBERT). The w/o Contrastive Pretraining variant is represented by comparison videos; a checkpoint for that variant is not included in this release.

Download and Use

Install the Hugging Face CLI if needed:

pip install -U huggingface_hub

Download all model and evaluation files (approximately 16.5 GB):

hf download kmraven/AudioLDM-ControlNet \
  --include "models/*" --include "testsets_video_results/*" \
  --local-dir data/releases/AudioLDM-ControlNet

To download only the full-model checkpoint and configuration:

hf download kmraven/AudioLDM-ControlNet \
  "models/full/checkpoints/checkpoint-global_step=299999.ckpt" \
  "models/full/audioldm_original_medium_stretch_pretrained.yaml" \
  --local-dir data/releases/AudioLDM-ControlNet

Run preprocessing and evaluation with the official source code and setup instructions. These are project-specific PyTorch Lightning checkpoints; use the repository's evaluation scripts. The YAML files specify the training settings and use relative paths. Dataset inputs and separately referenced upstream weights must be prepared as described in the source repository. This release does not include a standalone Stage 1 model_best.pth checkpoint.

Exact training resume, including optimizer state and global step, may be incompatible with the current training code. If exact resume fails, the source repository recommends commit 0c1f4e3197a7c339586ce5769c6a4510dfbc6f30.

Checkpoint Details

Framework: PyTorch / PyTorch Lightning
Checkpoint format: .ckpt
Paper variants with checkpoints: Full model and w/o MotionBERT
Training step: 299999
Audio sampling rate: 16 kHz
Motion input frame rate: 60 FPS
Motion sequence length used by the paper configuration: 128 frames

The checkpoint contains a PyTorch Lightning-compatible state_dict.

Input Representation

The model is conditioned on human pose sequences represented as NumPy arrays with the following shape:

[frames, 17, 3]

The 17 joints follow COCO joint order. The final dimension contains the per-joint coordinates expected by the official preprocessing and evaluation code.

Training Procedure

Training is performed in two stages:

MotionBERT and MERT features are aligned using beat-guided contrastive pretraining based on BeatDance.
The pretrained dance encoder conditions an AudioLDM backbone through a ControlNet adapter.

The implementation, preprocessing code, training scripts, and evaluation scripts are available in the official GitHub repository.

Citation

@article{kimura2026dance,
  title={Dance to Music Generation leveraging Pre-training with Unpaired Data and Contrastive Alignment},
  author={Kimura, Ryota and Park, Sangheon and Polouliakh, Natalia and Akama, Taketo},
  journal={arXiv preprint arXiv:2607.10537},
  year={2026}
}

Acknowledgements

This project builds on AudioLDM, ControlNet, MotionBERT, MERT, and BeatDance. Please cite the corresponding projects in addition to this work when using the model.

License

The source code repository is distributed under the MIT License. The model checkpoint may also be subject to the licenses and terms of the upstream pretrained models and datasets. Review the applicable upstream licenses before redistribution or commercial use.

Fuentes

1 fuente

Hugging FacePrincipalVersión 81505b7066e04df833db794e856f51de996cdece

huggingface.co/kmraven/AudioLDM-ControlNet/tree/81505b7066e04df833db794e856f51de996cdece

Verificado 8 oct

Artefactos del modelo

3 artefactos

checkpoint-global_step=299999.ckpt

ckpt · 7,48 GB · SHA-256 5673a101b25c…b239 · Hugging Face

Descargar

models/full/checkpoints/checkpoint-global_step=299999.ckpt

ckpt · 7,48 GB · SHA-256 5673a101b25c…b239 · Hugging Face

Descargar

models/without_motioinbert/checkpoints/checkpoint-global_step=299999.ckpt

ckpt · 7,48 GB · SHA-256 9bd26513abe4…905f · Hugging Face

Descargar

Extractos de fuentes

3 extractos

Readme8 octhuggingface.co/kmraven/AudioLDM-ControlNet

--- license: other library_name: pytorch tags: - audio-generation - music-generation - dance-to-music - motion-conditioned-generation - controlnet - audioldm - pytorch-lightning - arxiv:2607.10537 --- # AudioLDM-ControlNet Official checkpoints and evaluation artifacts for the paper: **Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment** - Paper: https://huggingface.co/papers/2607.10537 - arXiv: https://arxiv.org/abs/2607.10537 - Source code: https://github.com/kmraven/AudioLDM-ControlNet ## Model Description AudioLDM-ControlNet is a dance-conditioned music generation model. It combines: 1. pretrained motion and music encoders, 2. beat-guided contrastive pretraining for cross-modal alignment, and 3. a ControlNet-style conditioning module built on an AudioLDM backbone. This repository includes the **full model**, the **w/o MotionBERT** checkpoint, along with generated audio, evaluation metrics, training logs, and comparison videos. ## Repository Contents Model checkpoints and evaluation outputs are organized under `models/` and `testsets_video_results/`. | Path | Contents | | --- | --- | | [`models/full/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/models/full) | Full-model checkpoint, YAML configuration, 36 generated WAV files and `metrics.txt` in `300ksteps/`, and the W&B training run in `wandb/` | | [`models/without_motioinbert/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/models/without_motioinbert) | w/o MotionBERT checkpoint, YAML configuration, and 36 generated WAV files and `metrics.txt` in `300ksteps/` | | [`testsets_video_results/`](https://huggingface.co/kmraven/AudioLDM-ControlNet/tree/main/testsets_video_results) | 216 MP4 comparison videos: 36 clips for each of the six conditions listed below | | `checkpoint-global_step=299999.ckpt` | Full-model checkpoint, also available under `models/full/checkpoints/` | | `audioldm_original_medium_stretch_pretrained.yaml` | Full-model configuration, also available under `models/full/` | The root-level checkpoint and configuration are identical to their counterparts under `models/full/`. Both checkpoints are named `checkpoint-global_step=299999.ckpt` and are stored in their respective variant's `checkpoints/` directory. The matching configurations are: - Full model: `models/full/audioldm_original_medium_stre...

Readme31 julhuggingface.co/kmraven/AudioLDM-ControlNet

--- license: other library_name: pytorch tags: - audio-generation - music-generation - dance-to-music - motion-conditioned-generation - controlnet - audioldm - pytorch-lightning - arxiv:2607.10537 --- # AudioLDM-ControlNet Official model checkpoint for the paper: **Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment** - Paper: https://huggingface.co/papers/2607.10537 - arXiv: https://arxiv.org/abs/2607.10537 - Source code: https://github.com/kmraven/AudioLDM-ControlNet ## Model Description AudioLDM-ControlNet is a dance-conditioned music generation model. It combines: 1. pretrained motion and music encoders, 2. beat-guided contrastive pretraining for cross-modal alignment, and 3. a ControlNet-style conditioning module built on an AudioLDM backbone. The released checkpoint corresponds to the **full model** reported in the paper. ## Repository Contents The following files are stored in the root of this model repository: | File | Description | | --- | --- | | `checkpoint-global_step=299999.ckpt` | Full AudioLDM-ControlNet model checkpoint at training step 299999 | | `audioldm_original_medium_stretch_pretrained.yaml` | Model and preprocessing configuration corresponding to the checkpoint | ## Checkpoint Details - Framework: PyTorch / PyTorch Lightning - Checkpoint format: `.ckpt` - Paper variant: Full model - Training step: 299999 - Audio sampling rate: 16 kHz - Motion input frame rate: 60 FPS - Motion sequence length used by the paper configuration: 128 frames The checkpoint contains a PyTorch Lightning-compatible `state_dict`. ## Input Representation The model is conditioned on human pose sequences represented as NumPy arrays with the following shape: ```text [frames, 17, 3] ``` The 17 joints follow COCO joint order. The final dimension contains the per-joint coordinates expected by the official preprocessing and evaluation code. ## Training Procedure Training is performed in two stages: 1. MotionBERT and MERT features are aligned using beat-guided contrastive pretraining based on BeatDance. 2. The pretrained dance encoder conditions an AudioLDM backbone through a ControlNet adapter. The implementation, preprocessing code, training scripts, and evaluation scripts are available in the official GitHub repository. ## Citation ```bibtex @article{kimura2026dance, title={Dance to Music Ge...

Note31 julhuggingface.co/kmraven/AudioLDM-ControlNet

Source context: 0 downloads · 0 likes · Library pytorch · Repo kmraven/AudioLDM-ControlNet