Human image animation has attracted increasing attention and developed rapidly due to its broad applications in digital humans. However, existing methods rely on 2D-rendered pose images for motion guidance, which limits generalization and discards essential 3D information.
To tackle these problems, we propose MTVCrafter (Motion Tokenization Video Crafter), the first framework that directly models raw 3D motion sequences for open-world human image animation beyond intermediate 2D representations.
We introduce 4DMoT (4D motion tokenizer) to encode raw motion data into discrete motion tokens, preserving 4D compact yet expressive spatio-temporal information.
Then, we propose MV-DiT (Motion-aware Video DiT), which integrates a motion attention module and 4D positional encodings to effectively modulate vision tokens with motion tokens.
The overall pipeline facilitates high-quality human video generation guided by 4D motion tokens.
MTVCrafter achieves state-of-the-art results with an FID-VID of 6.98, outperforming the second-best by approximately 65%. It generalizes well to diverse characters (single/multiple, full/half-body) across various styles.
🎯 Motivation
Motivation
Our motivation is that directly tokenizing 4D motion captures more faithful and expressive information than traditional 2D-rendered pose images derived from the driven video.
💡 Method
Method
(1) 4DMoT:
Our 4D motion tokenizer consists of an encoder-decoder framework to learn spatio-temporal latent representations of SMPL motion sequences,
and a vector quantizer to learn discrete tokens in a unified space.
All operations are performed in 2D space along frame and joint axes.
Method
(2) MV-DiT:
Based on video DiT architecture,
we design a 4D motion attention module to combine motion tokens with vision tokens.
Since the tokenization and flattening disrupted positional information,
we introduce 4D RoPE to recover the spatio-temporal relationships.
To further improve the quality of generation and generalization,
we use learnable unconditional tokens for motion classifier-free guidance.
🛠️ Installation
We recommend using a clean Python environment (Python 3.10+).
(Optional but recommended)
Download the enhanced LoRA for better performance of Wan2.1_I2V_14B: Wan2.1_I2V_14B_FusionX_LoRA.safetensors
Place it under the wan2.1/ folder.
🚀 Usage
To animate a human image with a given 3D motion sequence,
you first need to prepare SMPL motion-video pairs. You can either:
Use the provided sample data: data/sampled_data.pkl, or
Extract SMPL motion sequences from your own driving video using:
python process_nlf.py "your_video_directory"
This will generate a motion-video .pkl file under "your_video_directory".
1python infer_17b.py \2 --ref_image_path "ref_images/woman.png"\3 --motion_data_path "data/sampled_data.pkl"\4 --output_path "inference_output"\5 --prompt "The woman is dancing on the beach, waves, sunset."
Arguments:
--ref_image_path: Path to the reference character image.
--motion_data_path: Path to the SMPL motion sequence (.pkl format).
--output_path: Directory to save the generated video.
--prompt (optional): Text prompt describing the scene or style.
🏋️♂️ Training 4DMoT
To train the 4DMoT tokenizer on your own dataset:
accelerate launch train_vqvae.py
💙 Acknowledgement
MTVCrafter is largely built upon
CogVideoX,
Wan-2-1-Fun.
We sincerely acknowledge these open-source codes and models.
We also appreciate the valuable insights from the researchers at Institute of Artificial Intelligence (TeleAI), China Telecom, and Shenzhen Institute of Advanced Technology.
📄 Citation
If you find our work useful, please consider citing:
bibtex
1@article{ding2025mtvcrafter,
2 title={MTVCrafter: 4D Motion Tokenization for Open-World Human Image Animation},
3 author={Ding, Yanbo and Hu, Xirui and Guo, Zhizhi and Zhang, Chi and Wang, Yali},
4 journal={arXiv preprint arXiv:2505.10238},
5 year={2025}
6}
📬 Contact
For questions or collaboration, feel free to reach out via GitHub Issues
or email me at 📧 yb.ding@siat.ac.cn.
