GeometryCrafter is still under active development!
We recommend that everyone use English to communicate on issues, as this helps developers from around the world discuss, share experiences, and answer questions together. For further implementation details, please contact xutx21@mails.tsinghua.edu.cn. For business licensing and other related inquiries, don't hesitate to contact wbhu@tencent.com.
If you find GeometryCrafter useful, please help ⭐ this repo, which is important to Open-Source projects. Thanks!
📝 Introduction
We present GeometryCrafter, a novel approach that estimates temporally consistent, high-quality point maps from open-world videos, facilitating downstream applications such as 3D/4D reconstruction and depth-based video editing or generation. This model is described in detail in the paper GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors.
Release Notes:
[01/04/2025] 🔥🔥🔥GeometryCrafter is released now, have fun!
Run inference code on our provided demo videos at 1.27FPS, which requires a GPU with ~40GB memory for 110 frames with 1024x576 resolution:
bash
1python run.py \2 --video_path examples/video1.mp4 \3 --save_folder workspace/examples_output \4 --height 576 --width 10245# resize the input video to the target resolution for processing, which should be divided by 64 6# the output point maps will be restored to the original resolution before saving7# you can use --downsample_ratio to downsample the input video or reduce --decode_chunk_size to save the memory usage
Run inference code with our deterministic variant at 1.50 FPS
To create the dataset we use in the paper, you need to run evaluation/preprocess/gen_{dataset_name}.py.
You need to change DATA_DIR and OUTPUT_DIR first accordint to your working environment.
Then you will get the preprocessed datasets containing extracted RGB video and point map npz files. We also provide the catelog of these files.
Inference for all datasets scripts:
bash evaluation/run_batch.sh
(Remember to replace the data_root_dir and save_root_dir with your path.)
Evaluation for all datasets scripts (scale-invariant point map estimation):
bash evaluation/eval.sh
(Remember to replace the pred_data_root_dir and gt_data_root_dir with your path.)
Evaluation for all datasets scripts (affine-invariant depth estimation):
bash evaluation/eval_depth.sh
(Remember to replace the pred_data_root_dir and gt_data_root_dir with your path.)
We also provide the comparison results of MoGe and the deterministic variant of our method. You can evaluate these methods under the same protocol by uncomment the corresponding lines in evaluation/run.shevaluation/eval.shevaluation/run_batch.sh and evaluation/eval_depth.sh.
🤝 Contributing
Welcome to open issues and pull requests.
Welcome to optimize the inference speed and memory usage, e.g., through model quantization, distillation, or other acceleration techniques.
📜 Citation
If you find this work helpful, please consider citing:
bibtex
1@misc{xu2025geometrycrafterconsistentgeometryestimation,
2 title={GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors},
3 author={Tian-Xing Xu and Xiangjun Gao and Wenbo Hu and Xiaoyu Li and Song-Hai Zhang and Ying Shan},
4 year={2025},
5 eprint={2504.01016},
6 archivePrefix={arXiv},
7 primaryClass={cs.GR},
8 url={https://arxiv.org/abs/2504.01016},
9}
