You can deploy this model and start earning money today!

yuxuanx/
gen3diffusion

safetensors

mit

Views

No views yet

Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy

This repository contains the Gen-3Diffusion model presented in the paper Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy.

Gen-3Diffusion addresses the challenging problem of creating realistic 3D objects and clothed avatars from a single RGB image. It leverages a pre-trained 2D diffusion model and a 3D diffusion model, synchronizing them at both training and sampling time. This synergy allows the 2D model to provide strong generalization for shapes, while the 3D model enhances multi-view consistency, leading to high-fidelity geometry and texture in generated 3D objects and avatars.

Key Insight :raised_hands:

2D foundation models are powerful but output lacks 3D consistency!
3D generative models can reconstruct 3D representation but is poor in generalization!
How to combine 2D foundation models with 3D generative models?:
- they are both diffusion-based generative models => Can be synchronized at each diffusion step
- 2D foundation model helps 3D generation => provides strong prior informations about 3D shape
- 3D representation guides 2D diffusion sampling => use rendered output from 3D reconstruction for reverse sampling, where 3D consistency is guaranteed

Pretrained Weights

Our pretrained weights can be downloaded from Hugging Face.

bash

mkdir checkpoints_obj && cd checkpoints_obj
wget https://huggingface.co/yuxuanx/gen3diffusion/resolve/main/model.safetensors
wget https://huggingface.co/yuxuanx/gen3diffusion/resolve/main/model_1.safetensors
wget https://huggingface.co/yuxuanx/gen3diffusion/resolve/main/pifuhd.pt
cd ..

The avatar reconstruction module is same to Human-3Diffusion. Please skip if you already installed Human-3Diffusion.

bash

mkdir checkpoints_avatar && cd checkpoints_avatar
wget https://huggingface.co/yuxuanx/human3diffusion/resolve/main/model.safetensors
wget https://huggingface.co/yuxuanx/human3diffusion/resolve/main/model_1.safetensors
wget https://huggingface.co/yuxuanx/human3diffusion/resolve/main/pifuhd.pt
cd ..

Sample Usage (Inference)

The following commands illustrate how to use the model for image-to-3D object and avatar generation. Please refer to the GitHub repository for full installation and setup instructions.

bash

# given one image of object, generate 3D-GS object
# subject should be centered in a square image, please crop properly 
# recenter plays a huge role in object reconstruction. Please adjust the recentering if the reconstruction doesn't work well
python infer.py --test_imgs test_imgs_obj --output output_obj --checkpoints checkpoints_obj

# given generated 3D-GS, perform TSDF mesh extraction
python infer_mesh.py --test_imgs test_imgs_obj --output output_obj --checkpoints checkpoints_obj --mesh_quality high

bash

# given one image of human, generate 3D-GS avatar
# subject should be centered in a square image, please crop properly
python infer.py --test_imgs test_imgs_avatar --output output_avatar --checkpoints checkpoints_avatar

# given generated 3D-GS, perform TSDF mesh extraction
python infer_mesh.py --test_imgs test_imgs_avatar --output output_avatar --checkpoints checkpoints_avatar --mesh_quality high

Citation :writing_hand:

If you find our work helpful or inspiring, please feel free to cite it:

bibtex

@inproceedings{xue2024gen3diffusion,
  title     = {{Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy }},
  author    = {Xue, Yuxuan and Xie, Xianghui and Marin, Riccardo and Pons-Moll, Gerard.},
  journal   = {Arxiv},
  year      = {2024},
}