Here are several example videos generated by RealisMotion.
Note that the GIFs shown here have some degree of visual quality degradation.
Please visit our project page for more than 100 videos examples.
Generating human videos with realistic and controllable motions is a challenging task. While existing methods can generate visually compelling videos, they lack separate control over four key video elements: foreground subject, background video, human trajectory and action patterns. In this paper, we propose a decomposed human motion control and video generation framework that explicitly decouples motion from appearance, subject from background, and action from trajectory, enabling flexible mix-and-match composition of these elements. Concretely, we first build a ground-aware 3D world coordinate system and perform motion editing directly in the 3D space. Trajectory control is implemented by unprojecting edited 2D trajectories into 3D with focal-length calibration and coordinate transformation, followed by speed alignment and orientation adjustment; actions are supplied by a motion bank or generated via text-to-motion methods. Then, based on modern text-to-video diffusion transformer models, we inject the subject as tokens for full attention, concatenate the background along the channel dimension, and add motion (trajectory and action) control signals by addition. Such a design opens up the possibility for us to generate realistic videos of anyone doing anything anywhere. Extensive experiments on benchmark datasets and real-world cases demonstrate that our method achieves state-of-the-art performance on both element-wise controllability and overall video quality.
We provide two versions for inference: the first is the text-to-video (T2V) version (same as the model in the paper), the second is the image-to-video (I2V) version (to avoid duplicate work, we directly combine with the concurrent work RealisDance-Dit).
Version Type
Advantage
Disadvantage
Text-to-Video (T2V) version
support fully decomposed subject and background control
CUDA_VISIBLE_DEVICES=0 python inference_t2v.py \
--ref_path input/demo/InfernalAffairs/0_input_video.mp4 \
--ref_cs_map_path input/demo/InfernalAffairs/cs_map.mp4 \
--ref_mask_path input/demo/InfernalAffairs/mask.mp4 \
--background_path input/demo/InfernalAffairs/0_input_video.mp4 \
--background_mask_path input/demo/InfernalAffairs/mask.mp4 \
--motion_folder input/demo/InfernalAffairs_falldown3_InfernalAffairs \
--ref_index 0 \
--background_start_index 0 \
--background_num_frames 97 \
--motion_start_index 0 \
--motion_num_frames 97 \
--prompt "A tense scene on a rooftop with a view of a bustling city and harbor, showcasing dramatic cityscape under a cloudy sky. Two figures are centered in the composition, one dressed in a dark outfit, extending their arm with a gun pointed at another figure wearing a navy suit. The urban backdrop features high-rise buildings and mountains in the distance, suggesting a metropolitan setting with a blend of natural elements." \
--save-dir ./outputs
I2V
python inference_i2v.py \
--ref inputs/example_video/justin.png \
--smpl inputs/demo/justin_tstageboy_justin/cs_map.mp4 \
--hamer inputs/demo/justin_tstageboy_justin/hamer.mp4 \
--prompt "Justin Bieber is walking along a rustic, dirt path in a rural landscape under an overcast sky, creating a moody and atmospheric scene. They are dressed in a roomy pink hoodie layered over a dark jacket, coupled with a long, flowing gray skirt that adds an eclectic touch to their ensemble. The outfit is completed with chunky black boots, providing contrast and robustness. The background features sparse buildings and expanses of dry grass, suggesting an isolated countryside setting, while the attire reflects a unique blend of casual and avant-garde fashion." \
--save-dir ./outputs
Note: add --enable-teacache to inference with TeaCache for acceleration (optional, may cause quality degradation); add --save-gpu-memory to inference with small GPU memory (optional, will be super slow. Can be used with TeaCache).
Inference with multi GPUs (Optional. Can be used with TeaCache)
Version Type
Bash Command
T2V
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --nproc_per_node=8 inference_t2v.py \
--ref_path input/demo/InfernalAffairs/0_input_video.mp4 \
--ref_cs_map_path input/demo/InfernalAffairs/cs_map.mp4 \
--ref_mask_path input/demo/InfernalAffairs/mask.mp4 \
--background_path input/demo/InfernalAffairs/0_input_video.mp4 \
--background_mask_path input/demo/InfernalAffairs/mask.mp4 \
--motion_folder input/demo/InfernalAffairs_falldown3_InfernalAffairs \
--ref_index 0 \
--background_start_index 0 \
--background_num_frames 97 \
--motion_start_index 0 \
--motion_num_frames 97 \
--prompt "A tense scene on a rooftop with a view of a bustling city and harbor, showcasing dramatic cityscape under a cloudy sky. Two figures are centered in the composition, one dressed in a dark outfit, extending their arm with a gun pointed at another figure wearing a navy suit. The urban backdrop features high-rise buildings and mountains in the distance, suggesting a metropolitan setting with a blend of natural elements." \
--save-dir ./outputs \
--multi-gpu
I2V
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --nproc_per_node=8 inference_i2v.py \
--ref inputs/example_video/justin.png \
--smpl inputs/demo/justin_tstageboy_justin/cs_map.mp4 \
--hamer inputs/demo/justin_tstageboy_justin/hamer.mp4 \
--prompt "Justin Bieber is walking along a rustic, dirt path in a rural landscape under an overcast sky, creating a moody and atmospheric scene. They are dressed in a roomy pink hoodie layered over a dark jacket, coupled with a long, flowing gray skirt that adds an eclectic touch to their ensemble. The outfit is completed with chunky black boots, providing contrast and robustness. The background features sparse buildings and expanses of dry grass, suggesting an isolated countryside setting, while the attire reflects a unique blend of casual and avant-garde fashion." \
--save-dir ./outputs \
--multi-gpu
5. Motion Editing and Guidance Condition Rendering
To edit the trajectory, orientation and action of human, please follow followng steps.
1. Setup Environment
First, please install GVHMR and DPVO following install GVHMR. The nvcc in third-party/DPVO/setup.py should be modified as ['-O3', '-gencode', 'arch=compute_90,code=sm_90'] for H20 GPUs.
Then, please install DepthPro for focal length calibration as follows (optional).
git clone https://github.com/apple/ml-depth-pro
cd ml-depth-pro
pip install .
source get_pretrained_models.sh
cd ..
2. SMPL-X Estimation
We first estimate the SMPL-X for the input foreground subject, background and motion videos / images.
cd RealisMotion
export PYTHONPATH="/mnt_video/jingyun.ljy/code/GVHMR/hmr4d:$PYTHONPATH"
# process foreground and background
python hmc/render_demo.py --video=inputs/example_video/internalaffairs.mp4 --output_root inputs/demo --track_id 1
# process motion
python hmc/render_demo.py --video=inputs/example_video/falldown.mp4 --output_root inputs/demo
By default, --track_id is set as 0 to track the first person. Use -s for static background. When you only have an image, turn it to a video first as below.
To edit the motion, you need to specify the background path, the motion path, and the reference foreground path. We currently provide four examples for different usecases.
For kid, add --kid 1.0. One can use a float number between 0 and 1 to interpolate between adult and kid.
The human mask and hamer (hand pose) are optional, but providing them could improve the video quality. To obtain the human mask, one can install MatAnyone locally or use this MatAnyone Online Demo. Without the human mask, we will extract one from the SMPL-X depth. To obtain the hamer, please refer to Hamer Preparation. Without hamer, we will use the standard hand pose in SMPL-X.
Disclaimer
This project is released for academic use.
We disclaim responsibility for user-generated content.
Citation
@article{liang2025realismotion,
title={RealisMotion: Decomposed Human Motion Control and Video Generation in the World Space},
author={Liang, Jingyun and Zhou, Jingkai and Li, Shikai and Cao, Chenjie and Sun, Lei and Qian, Yichen and Chen, Weihua and Wang, Fan},
journal={arXiv preprint arXiv:2508.08588},
year={2025}
}
Acknowledgement
We thank the authors of WHAM, 4D-Humans, and ViTPose-Pytorch for their great works, without which our project/code would not be possible.
