This model is part of the Wavelet Latent Diffusion (WaLa) paper, capable of generating high-quality 3D shapes from point cloud inputs with detailed geometry and complex structures.
Model Details
Model Description
WaLa-PC-1B is a large-scale 3D generative model trained on a massive dataset of over 10 million publicly-available 3D shapes. It can efficiently generate a wide range of high-quality 3D shapes from point cloud inputs in just 2-4 seconds. The model uses a wavelet-based compact latent encoding and a billion-parameter architecture to achieve superior performance in terms of geometric detail and structural plausibility.
Developed by: Aditya Sanghi, Aliasghar Khani, Chinthala Pradyumna Reddy, Arianna Rampini, Derek Cheung, Kamal Rahimi Malekshan, Kanika Madan, Hooman Shayani
This model is specifically developed for the sole purpose of scientific research and development. It is intended for academic and research purposes only for the theoretical exploration and demonstration of the WaLa 3D generative framework. Please see here for inferencing instructions.
Out-of-Scope Use
The model should not be used for:
Commercial purposes
Creation of load-bearing physical objects the failure of which could cause property damage or personal injury
Any usage not in compliance with the license, in particular, the "Acceptable Use" section.
Bias, Risks, and Limitations
Bias
The model may inherit biases present in the publicly-available training datasets, which could lead to uneven representation of certain object types or styles.
The model's performance may degrade for object categories or styles that are underrepresented in the training data.
Risks and Limitations
The quality of the generated 3D output may be impacted by the quality and completeness of the input point cloud.
The model may occasionally generate implausible shapes, especially when the input point cloud is sparse or noisy. Even theoretically plausible shapes should not be relied upon for real-world structural soundness.
The model was trained on a dataset of over 10 million 3D shapes aggregated from 19 different publicly-available sub-datasets, including ModelNet, ShapeNet, SMLP, Thingi10K, SMAL, COMA, House3D, ABC, Fusion 360, 3D-FUTURE, BuildingNet, DeformingThings4D, FG3D, Toys4K, ABO, Infinigen, Objaverse, and two subsets of ObjaverseXL (Thingiverse and GitHub).
Training Procedure
Preprocessing
Each 3D shape in the dataset was converted into a truncated signed distance function (TSDF) with a resolution of 256³. The TSDF was then decomposed using a discrete wavelet transform to create the wavelet-tree representation used by the model. For point cloud conditioning, 2,500 points were randomly selected from each shape.
Training Hyperparameters
Training regime: Please refer to the paper.
Speeds, Sizes, Times
The model contains approximately 967 million parameters.
The model can generate shapes within 2-4 seconds.
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was evaluated on the Google Scanned Objects (GSO) dataset and a validation set from the training data (MAS validation data).
Factors
The evaluation considered various factors such as the quality of generated shapes, the ability to capture fine details and complex structures, and the model's performance across different object categories.
Metrics
The model was evaluated using the following metrics:
Intersection over Union (IoU)
Light Field Distance (LFD)
Chamfer Distance (CD)
Results
The point cloud to 3D model achieved the following results on the GSO dataset:
LFD: 1114.01
IoU: 0.9389
CD: 0.0011
On the MAS validation dataset:
LFD: 1467.55
IoU: 0.8625
CD: 0.0014
Technical Specifications
Model Architecture and Objective
The model uses a U-ViT architecture with modifications. It employs a wavelet-based compact latent encoding to effectively capture both coarse and fine details of 3D shapes from point cloud inputs. The input points are encoded using a PointNet encoder, followed by attention pooling to aggregate the feature vectors into condition latent vectors.
Compute Infrastructure
Hardware
The model was trained on NVIDIA H100 GPUs.
Citation
@misc{sanghi2024waveletlatentdiffusionwala,
title={Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings},
author={Aditya Sanghi and Aliasghar Khani and Pradyumna Reddy and Arianna Rampini and Derek Cheung and Kamal Rahimi Malekshan and Kanika Madan and Hooman Shayani},
year={2024},
eprint={2411.08017},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2411.08017},
}
