[Jun, 2025] Check out our new models: SpatialLM1.1-Llama-1B and SpatialLM1.1-Qwen-0.5B, now available on Hugging Face. SpatialLM1.1 doubles the point cloud resolution, incorporates a more powerful point cloud encoder Sonata and supports detection with user-specified categories.
SpatialLM is a 3D large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object bounding boxes with their semantic categories. Unlike previous methods that require specialized equipment for data collection, SpatialLM can handle point clouds from diverse sources such as monocular video sequences, RGBD images, and LiDAR sensors. This multimodal architecture effectively bridges the gap between unstructured 3D geometric data and structured 3D representations, offering high-level semantic understanding. It enhances spatial reasoning capabilities for applications in embodied robotics, autonomous navigation, and other complex 3D scene analysis tasks.
SpatialLM reconstructs 3D layout from a monocular RGB video with MASt3R-SLAM. Results aligned to video with GT cameras for visualization.
1# clone the repository2git clone https://github.com/manycore-research/SpatialLM.git
3cd SpatialLM
45# create a conda environment with cuda 12.46conda create -n spatiallm python=3.117conda activate spatiallm
8conda install -y nvidia/label/cuda-12.4.0::cuda-toolkit conda-forge::sparsehash
910# Install dependencies with poetry11pip install poetry && poetry config virtualenvs.create false --local
12poetry install13# SpatialLM1.0 dependency14poe install-torchsparse # Building wheel for torchsparse will take a while15# SpatialLM1.1 dependency16poe install-sonata # Building wheel for flash-attn will take a while
Inference
In the current version of SpatialLM, input point clouds are considered axis-aligned where the z-axis is the up axis. This orientation is crucial for maintaining consistency in spatial understanding and scene interpretation across different datasets and applications.
Example preprocessed point clouds, reconstructed from RGB videos using MASt3R-SLAM, are available in SpatialLM-Testset.
For tasks that include object box estimation, you can specify a subset of the 59 furniture categories, and the model will only predict objects within those specified categories. For example:
Use rerun to visualize the point cloud and the predicted structured 3D layout output:
bash
1# Convert the predicted layout to Rerun format2python visualize.py --point_cloud pcd/scene0000_00.ply --layout scene0000_00.txt --save scene0000_00.rrd
34# Visualize the point cloud and the predicted layout5rerun scene0000_00.rrd
Evaluation
To evaluate the performance of SpatialLM, we provide eval.py script that reports the benchmark results on the SpatialLM-Testset in the table below in section Benchmark Results.
1# Run inference on the PLY point clouds in folder SpatialLM-Testset/pcd with SpatialLM1.1-Qwen-0.5B model2python inference.py --point_cloud SpatialLM-Testset/pcd --output SpatialLM-Testset/pred --model_path manycore-research/SpatialLM1.1-Qwen-0.5B
34# Evaluate the predicted layouts5python eval.py --metadata SpatialLM-Testset/test.csv --gt_dir SpatialLM-Testset/layout --pred_dir SpatialLM-Testset/pred --label_mapping SpatialLM-Testset/benchmark_categories.tsv
Example using a custom video
We provide an example of how to use our model to estimate scene layout starting from a RGB video with the newly released SLAM3R in EXAMPLE.md. These steps work for MASt3R-SLAM, and other reconstruction methods as well.
SpatialLM Dataset
The SpatialLM dataset is a large-scale, high-quality synthetic dataset designed by professional 3D designers and used for real-world production. It contains point clouds from 12,328 diverse indoor scenes comprising 54,778 rooms, each paired with rich ground-truth 3D annotations. SpatialLM dataset provides an additional valuable resource for advancing research in indoor scene understanding, 3D perception, and related applications.
For access to photorealistic RGB/Depth/Normal/Semantic/Instance panoramic renderings and camera trajectories used to generate the SpatialLM point clouds, please refer to the SpatialGen project for more details.
We provide a test set of 107 preprocessed point clouds, reconstructed from RGB videos using MASt3R-SLAM. SpatialLM-Testset is quite challenging compared to prior clean RGBD scans datasets due to the noises and occlusions in the point clouds reconstructed from monocular RGB videos.
Layout estimation focuses on predicting architectural elements, i.e., walls, doors, and windows, within an indoor scene. We evaluated this task on the Structured3D dataset. For RoomFormer, we directly downloaded the model checkpoint. SceneScript and SpatialLM were first trained on our dataset, and further fine-tuned on Structured3D.
Method
RoomFormer
SceneScript (finetuned)
SpatialLM1.1-Qwen-0.5B (finetuned)
F1 @.25 IoU
83.4
90.4
94.3
F1 @.5 IoU
81.4
89.2
93.5
3D Object Detection
We evaluate 3D object detection on ScanNet with annotations of 18 object categories. For V-DETR, we directly download the model checkpoint. SceneScript and SpatialLM were first trained on our dataset, and further fine-tuned on ScanNet.
Method
V-DETR
SceneScript (finetuned)
SpatialLM1.1-Qwen-0.5B (finetuned)
F1 @.25 IoU
65.1
49.1
65.6
F1 @.5 IoU
56.8
36.8
52.6
Zero-shot Detection on Videos
Zero-shot detection results on the challenging SpatialLM-Testset are reported in the following table:
SpatialLM-Llama-1B is derived from Llama3.2-1B-Instruct, which is licensed under the Llama3.2 license.
SpatialLM-Qwen-0.5B is derived from the Qwen-2.5 series, originally licensed under the Apache 2.0 License.
SpatialLM1.0 are built upon the SceneScript point cloud encoder, licensed under the CC-BY-NC-4.0 License. TorchSparse, utilized in this project, is licensed under the MIT License.
SpatialLM1.1 are built upon Sonata point cloud encoder, model weight is licensed under the CC-BY-NC-4.0 License. Code built on Pointcept is licensed under the Apache 2.0 License.
Citation
If you find this work useful, please consider citing:
bibtex
1@inproceedings{SpatialLM,
2 title = {SpatialLM: Training Large Language Models for Structured Indoor Modeling},
3 author = {Mao, Yongsen and Zhong, Junhao and Fang, Chuan and Zheng, Jia and Tang, Rui and Zhu, Hao and Tan, Ping and Zhou, Zihan},
4 booktitle = {Advances in Neural Information Processing Systems},
5 year = {2025}
6}
Acknowledgements
We would like to thank the following projects that made this work possible:
