Actor-critic music-to-dance generation with a learned choreographic memory.

Paper | Project Page | Original GitHub | Motius Checkpoint | Task Protocol

Bailando is the CVPR 2022 oral work Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory. It learns upper- and lower-body VQ codebooks, then composes those dance units autoregressively from music. The Motius release implements the model, audio frontend, AIST++ dataset, pipeline, representation bridge, and evaluator without importing an external checkout at runtime.

The previews are distinct AIST++ evaluation outputs rendered as neutral SMPL meshes at 512x512 and 30 fps. Their position-IK fit errors are 13.75, 13.89, and 13.71 mm. MP4 sources and fit reports are stored beside the GIF assets.

The Hugging Face artifact contains both VQ-VAE branches and the actor-critic GPT as safetensors, plus the complete architecture config, source hashes, license, and attribution. It does not require an upstream repository or a second checkpoint download.

Install the music frontend dependencies:

Generate from an audio file:

For exact benchmark reproduction, pass the released 438D AIST++ feature stream and paired initial motion. Only the first upper/lower VQ token initializes the generation, matching the official script:

Without initial_motion, the public demo seed (423, 12) is used. Raw-audio inference is a convenience path; use released precomputed features when exact paper parity across audio-library versions matters.

Motius ran the converted official epoch-500 VQ-VAE and epoch-10 GPT on all 40 cross-modal validation/test cases. FID and diversity use the 1,320 valid motion PKLs in the AIST++ v1 reference archive. Generated features use the first 1,200 frames; reference features use complete sequences; BeatAlign uses the complete generated sequence and the paired 60 fps music-beat stream.

Lower is better for FID, higher is better for BeatAlign, and diversity is interpreted relative to GT. The paper values are shown as parity targets and are not copied into the reproduced row.

These Motius joint-level diagnostics use the common SMPL-22 subset. They are not metrics from the Bailando paper.

Dynamic is an expressiveness statistic to compare with GT rather than minimize. The floor-dependent diagnostics are reported in native metric units.

The public representation name is aistpp_smpl24_joints, shape (T,24,3) in metres with Y up. Joints 0:22 are the standard SMPL body chain and convert exactly to smpl22_joints:

Conversion to motion135 and SMPL mesh uses position IK because the generated tensor stores joint positions rather than local rotations. The three preview reports expose the resulting fit errors instead of hiding this lossy step.

The model code remains under the upstream S-Lab License 1.0. AIST++ annotations are CC BY 4.0. See the artifact LICENSE and ATTRIBUTIONS.md before redistribution or commercial use.