Views
No views yet
 |  |
 |  |
 |  |
Diffusion models have shown remarkable success in video generation. However, whether such models are truly aware of the 3D structure underlying visual observations, rather than simply reproducing plausible 2D projections, remains an open question. In this work, we investigate this question through human motion control, a task that requires precise modelling of 3D human geometry, motion, camera viewpoint, and scene context. Unlike prior methods that rely on rendered 2D motion guidance videos, we propose a render-free framework that conditions video generation directly on compressed 3D human mesh tokens. This representation preserves full 3D geometric information while enabling a unified token-based generation pipeline that processes video tokens jointly with motion tokens in a DiT-based architecture. This design requires the model to reason jointly about appearance, 3D structure, and camera viewpoint during video generation. Experimental results demonstrate strong performance on human motion control benchmarks, while reducing artifacts induced by view-dependent 2D guidance and trajectory-pose mismatches during editing. These findings suggest that video diffusion models, when equipped with mesh tokenization, can better capture complex 3D human structures and their interactions with the surrounding environment.
1git clone https://github.com/jingyunliang/MeshToken.git
2cd MeshToken
3
4conda create -n MeshToken python=3.10
5conda activate MeshToken
6
7pip install -r requirements.txt
8
9# install FA3
10git clone https://github.com/Dao-AILab/flash-attention.git
11cd flash-attention
12git checkout 0dfb28174333d9eefb7c1dd4292690a8458d1e89 # Important: using other FA3 might yield bad results on H20 GPUs
13cd hopper
14python setup.py install
15
16cd ../../HF_ENDPOINT=https://hf-mirror.com huggingface-cli xxxx if you need to speed up downloading. By the way, put pretrained_models/ under a fast disk path (e.g., /tmp/) can reduce the model loading time significantly. Later, you can load the model by --ckpt /tmp/pretrained_models/MeshToken in that case.huggingface-cli download --resume-download jingyunliang/MeshToken --local-dir pretrained_models/MeshTokennvcc in third-party/DPVO/setup.py should be modified as ['-O3', '-gencode', 'arch=compute_90,code=sm_90'] for H20 GPUs. Please symlink the checkpoints by ln -s YOUR_PATH/GVHMR/inputs/checkpoints inputs/checkpoints.git clone https://github.com/apple/ml-depth-pro
cd ml-depth-pro
pip install .
source get_pretrained_models.sh
cd .../third-party/Mesh-VQ-VAE.git clone https://github.com/g-fiche/Mesh-VQ-VAE.git ./third-party/Mesh-VQ-VAEexport PYTHONPATH="YOUR_PATH/GVHMR/hmr4d:$PYTHONPATH"
CUDA_VISIBLE_DEVICES=0 python inference_mt.py \
--background_path inputs/demo/image.png \
--motion_folder inputs/demo \
--motion_start_index 0 \
--motion_num_frames 97 \
--prompt "A tense scene on a rooftop with a view of a bustling city and harbor, showcasing dramatic cityscape under a cloudy sky. Two figures are centered in the composition, one dressed in a dark outfit, extending their arm with a gun pointed at another figure wearing a navy suit. The urban backdrop features high-rise buildings and mountains in the distance, suggesting a metropolitan setting with a blend of natural elements." \
--save-dir ./outputs \
--ckpt pretrained_models/MeshToken
--enable-teacache to inference with TeaCache for acceleration (optional, may cause quality degradation); add --save-gpu-memory to inference with small GPU memory (optional, will be super slow. Can be used with TeaCache).export PYTHONPATH="YOUR_PATH/GVHMR/hmr4d:$PYTHONPATH"
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --nproc_per_node=8 inference_mt.py \
--background_path inputs/demo/image.png \
--motion_folder inputs/demo \
--motion_start_index 0 \
--motion_num_frames 97 \
--prompt "A woman in black athletic wear runs along a paved seaside path, wearing white headphones. The path is lined with evenly spaced concrete pillars and metal railings, leading her past them. The ocean waves crash against the rocks in the background, under a clear blue sky with scattered white clouds. The scene is serene, with the woman focused on her run, the calm sea extending to the horizon, and the gentle motion of the waves adding to the tranquility of the setting." \
--save-dir ./outputs \
--multi-gpu \
--ckpt pretrained_models/MeshToken
@article{liang2026towards,
title={Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization},
author={Liang, Jingyun and Wei, Min and Li, Shikai and Han, Yizeng and Yuan, Hangjie and Sun, Lei and Chen, Weihua and Wang, Fan},
journal={arXiv preprint arXiv:2605.00000},
year={2026}
}