Model Card for Make-A-Shape Multi-View to 3D Model
This model is part of the Make-A-Shape paper, capable of generating high-quality 3D shapes from multi-view images with intricate geometric details, realistic structures, and complex topologies.
Model Details
Model Description
Make-A-Shape is a novel 3D generative framework trained on an extensive dataset of over 10 million publicly-available 3D shapes. The multi-view to 3D model is one of the conditional generation models in this framework. It can efficiently generate a wide range of high-quality 3D shapes from four view-specific images as inputs. The model uses a wavelet-tree representation and adaptive training strategy to achieve superior performance in terms of geometric detail and structural plausibility.
- Developed by: Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, Chi-Wing Fu
- Model type: 3D Generative Model
- License: Autodesk Non-Commercial (3D Generative) v1.0
For more information please look at the
Project Page and
the ICML paper.
Model Sources
Uses
Direct Use
This model is specifically developed for the sole purpose of scientific research and development. It is intended for academic and research purposes only for the theoretical exploration and demonstration of the Make-a-Shape 3D generative framework. Please see
here for inferencing instructions.
Out-of-Scope Use
The model should not be used for:
-
Commercial purposes
-
Creation of load-bearing physical objects the failure of which could cause property damage or personal injury
-
Any usage not in compliance with the
license, in particular, the "Acceptable Use" section.
Bias, Risks, and Limitations
Bias
-
The model may inherit biases present in the publicly-available training datasets, which could lead to uneven representation of certain object types or styles.
-
The model's performance may degrade for object categories or styles that are underrepresented in the training data.
Risks and Limitations
-
The quality of the generated 3D output may be impacted by the quality and clarity of the input image.
-
The model may occasionally generate implausible shapes, especially when the input image is ambiguous or of low quality. Even theoretically plausible shapes should not be relied upon for real-world structural soundness.
How to Get Started with the Model
Please refer to the instructions
here.
Training Details
Training Data
The model was trained on a dataset of over 10 million 3D shapes aggregated from 18 different publicly-available sub-datasets, including ModelNet, ShapeNet, SMPL, Thingi10K, SMAL, COMA, House3D, ABC, Fusion 360, 3D-FUTURE, BuildingNet, DeformingThings4D, FG3D, Toys4K, ABO, Infinigen, Objaverse, and two subsets of ObjaverseXL (Thingiverse and GitHub).
Training Procedure
Preprocessing
Each 3D shape in the dataset was converted into a truncated signed distance function (TSDF) with a resolution of 256³. The TSDF was then decomposed using a discrete wavelet transform to create the wavelet-tree representation used by the model.
Training Hyperparameters
- Training regime: Please refer to the paper.
Speeds, Sizes, Times
- The model was trained on 48 × A10G GPUs for about 20 days, amounting to around 23,000 GPU hours.
- The model can generate shapes within two seconds for most conditions.
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was evaluated on a test set consisting of 2% of the shapes from each sub-dataset in the training data, as well as on the entire Google Scanned Objects (GSO) dataset, which was not part of the training data.
Factors
The evaluation considered various factors such as the quality of generated shapes, the ability to capture fine details and complex structures, and the model's performance across different object categories.
Metrics
The model was evaluated using the following metrics:
- Intersection over Union (IoU)
- Light Field Distance (LFD)
- Chamfer Distance (CD)
Results
The multi-view to 3D model achieved the following results on the "Our Val" dataset:
- LFD: 2217.25
- IoU: 0.6707
- CD: 0.00350
On the GSO dataset:
- LFD: 1890.85
- IoU: 0.7460
- CD: 0.00337
Technical Specifications
Model Architecture and Objective
The model uses a U-ViT architecture with learnable skip-connections between the convolution and deconvolution blocks. It employs a wavelet-tree representation and a subband adaptive training strategy to effectively capture both coarse and fine details of 3D shapes.
Compute Infrastructure
Hardware
The model was trained on 48 × A10G GPUs.
Citation
BibTeX:
1@InProceedings{pmlr-v235-hui24a,
2 title = {Make-A-Shape: a Ten-Million-scale 3{D} Shape Model},
3 author = {Hui, Ka-Hei and Sanghi, Aditya and Rampini, Arianna and Rahimi Malekshan, Kamal and Liu, Zhengzhe and Shayani, Hooman and Fu, Chi-Wing},
4 booktitle = {Proceedings of the 41st International Conference on Machine Learning},
5 pages = {20660--20681},
6 year = {2024},
7 editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix},
8 volume = {235},
9 series = {Proceedings of Machine Learning Research},
10 month = {21--27 Jul},
11 publisher = {PMLR},
12 pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/hui24a/hui24a.pdf},
13 url = {https://proceedings.mlr.press/v235/hui24a.html},
14}
