You can deploy this model and start earning money today!

chenwang/
geco

safetensors

Views

No views yet

GECO🦎: Generative Image-to-3D within a SECOnd

Cleaned version, please open an issue if you found any problems while running.

Project Page | Code

Abstract: 3D generation has seen remarkable progress in recent years. Existing techniques, such as score distillation methods, produce notable results but require extensive per-scene optimization, impacting time efficiency. Alternatively, reconstruction-based approaches prioritize efficiency but compromise the quality due to their limited handling of uncertainty. We introduce GECO, a novel method for high-quality 3D generative modeling that operates within a second. Our approach addresses the prevalent issues of uncertainty and inefficiency in current methods through a two-stage approach. In the initial stage, we train a single-step multi-view generative model with score distillation. Then, a second-stage distillation is applied to address the challenge of view inconsistency from the multi-view prediction. This two-stage process ensures a balanced approach to 3D generation, optimizing both quality and efficiency. Our comprehensive experiments demonstrate that GECO achieves high-quality image-to-3D generation with an unprecedented level of efficiency.

Installation

bash

python3.10 -m venv GECO
source ./GECO/bin/activate
pip install -r requirements.txt

Inference

Download checkpoints:

bash

mkdir ./ckpts && cd ./ckpts
pip install huggingface_hub
python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='chenwang/geco', filename='model.safetensors', local_dir='.')"
python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='chenwang/geco', filename='network-snapshot-005000.pkl', local_dir='.')"

Or with the hf CLI:

bash

mkdir ./ckpts && cd ./ckpts
hf download chenwang/geco model.safetensors network-snapshot-005000.pkl --local-dir .

Then run:

python infer.py big --pipeline zero123plus1step --generate_mv --num_input_views 6 --bg 1.0 --cond_t 0 --init_t 950 --joint --config_path configs/instant-mesh-base.yaml --zero123plus_v 'v11' --fovy 50 --distance 2.5 --resume_pkl ./ckpts/network-snapshot-005000.pkl --resume ./ckpts/model.safetensors --output_dir ./results/test --test_path ./examples/animal_character.png

Training

First stage training:

cd first-stage; bash train.sh

Generating data for second stage training:

python sample_data.py --sample_bs 64 --sample_data_path YOUR_DATASET --sample_start 0 --sample_end 40000 --sample_output_path SAVED_DATA_PATH

python sample_data_onestep.py --sample_bs 64 --sample_data_path SAVED_DATA_PATH --sample_start 0 --sample_end 40000

Second stage training:

accelerate launch --config_file acc_configs/gpu4.yaml main.py big --workspace logs/include-input/ --num_input_views 6 --num_views 10 --num_epochs 5 --bg 1.0 --batch_size 1 --lr 1e-6 --save_freq 1 --eval_freq 1 --stage 2 --data_path SAVED_DATA_PATH --config_path configs/instant-mesh-base.yaml --zero123plus_v 'v11' --mixed_precision 'no'

Citation

If you consider our paper or code useful, please cite our paper:

@article{wang2024geco,
  title={GECO: Generative Image-to-3D within a Second},
  author={Wang, Chen and Gu, Jiatao and Long, Xiaoxiao and Liu, Yuan and Liu, Lingjie},
  year={2025}
}

Credit

Our code is build upon from InstantMesh, Zero123Plus and LGM. Thanks the authors for opensourcing.