NVPanoptix-3D is a 3D Panoptic Reconstruction model that reconstructs complete 3D indoor scenes from single RGB images, simultaneously performing 2D panoptic segmentation, depth estimation, 3D scene reconstruction, and 3D panoptic segmentation. Built upon Uni-3D (ICCV 2023) baseline architecture, this model enhances 3D understanding by replacing the backbone with VGGT (Visual Geometry Grounded Transformer) and integrating multi-plane occupancy-aware lifting from BUOL (CVPR 2023) for improved 3D scene re-projection. The model reconstructs complete 3D scenes with both object instances (things) and scene layout (stuff) in a unified framework. This model was trained on the 3D-FRONT and Matterport3D datasets.
This model is intended for researchers and developers building 3D scene understanding applications for indoor environments, including robotics navigation, augmented reality, virtual reality, and architectural visualization.
panoptic_seg_3d: [Batch, 256, 256, 256] - 3D panoptic segmentation with instance IDs
semantic_seg_3d: [Batch, 256, 256, 256] - 3D semantic segmentation with class labels
instance_info: List of dictionaries containing per-instance 3D meshes and metadata
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Optimized for NVIDIA A100 80GB GPUs (Ampere architecture).
Requires GPU with high memory capacity (≥40GB recommended).
Compatible with other NVIDIA Ampere or NVIDIA Hopper GPUs (e.g., H100), though memory and interconnect bandwidth may affect performance.
Preferred/Supported Operating System(s):
Preferred: Ubuntu 22.04.5 LTS (Jammy Jellyfish), tested with CUDA 11.8.
Supported: Other Ubuntu versions (20.04+, 22.04+) and Linux distributions with compatible CUDA 11.x drivers.
The model requires NVIDIA GPU with ≥40GB memory for training and ≥30GB for inference. By leveraging NVIDIA hardware (GPU cores) and software frameworks (CUDA libraries), the model achieves efficient training and inference. The integration of this model into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment.
Model Version(s)
1.0
(Pre-trained Panoptic Recon 3D model deployable to Triton Inference Server for inference)
Training, Testing, and Evaluation Datasets
Dataset Overview
Total Number of Datasets: 02 Datasets (3D-FRONT and Matterport3D)
Data Modality: Image, 3D Geometry.
3D-FRONT
Link:https://tianchi.aliyun.com/dataset/65347 Data Modality: Image, 3D Geometry Image Training Data Size: Less than a Million Images Data Collection Method: Synthetic - Photorealistic rendered images from CAD models Labeling Method: Synthetic Properties: 3D-FRONT is a synthetic dataset of indoor scenes featuring photorealistically rendered RGB images accompanied by ground-truth 3D geometry, depth maps, semantic labels, and instance segmentations. It encompasses a diverse range of room types—including bedrooms, living rooms, dining rooms, and offices—with realistic furniture arrangements representative of residential spaces. The dataset contains over 18,797 indoor scenes, each captured from multiple viewpoints, and is split into 4,389, 489, and 1,206 images for training, validation, and testing, respectively.
Matterport3D
Link:https://niessner.github.io/Matterport/ Data Modality: Image, 3D Geometry Image Training Data Size: Less than a Million Images Data Collection Method: Automatic/Sensors - Real-world 3D scans using Matterport Pro camera Labeling Method: Hybrid: Automatic/Sensors, Human - Semi-automatic with human verification Properties: Matterport3D is a real-world dataset comprising 3D reconstructions of 90 indoor scenes. It provides RGB images, depth maps, camera poses, and semantic annotations across diverse environments such as homes, offices, and other building types. Each scene includes dense 3D point clouds and surface reconstructions annotated with category-level semantic labels. The dataset is divided into 34,737, 4,898, and 8,631 images for training, validation, and testing, corresponding to 61, 11, and 18 scenes, respectively.
Inference
Acceleration Engine: Triton Test Hardware:
1x NVIDIA A100 80GB
1x NVIDIA H100 80GB
Configuration:
Precision: FP32
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.
