SpatialLM is a 3D large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object bounding boxes with their semantic categories. Unlike previous methods that require specialized equipment for data collection, SpatialLM can handle point clouds from diverse sources such as monocular video sequences, RGBD images, and LiDAR sensors. This multimodal architecture effectively bridges the gap between unstructured 3D geometric data and structured 3D representations, offering high-level semantic understanding. It enhances spatial reasoning capabilities for applications in embodied robotics, autonomous navigation, and other complex 3D scene analysis tasks.
SpatialLM reconstructs 3D layout from a monocular RGB video with MASt3R-SLAM. Results aligned to video with GT cameras for visualization.
1# clone the repository2git clone https://github.com/manycore-research/SpatialLM.git
3cd SpatialLM
45# create a conda environment with cuda 12.46conda create -n spatiallm python=3.117conda activate spatiallm
8conda install -y nvidia/label/cuda-12.4.0::cuda-toolkit conda-forge::sparsehash
910# Install dependencies with poetry11pip install poetry && poetry config virtualenvs.create false --local
12poetry install13poe install-torchsparse # Building wheel for torchsparse will take a while
Inference
In the current version of SpatialLM, input point clouds are considered axis-aligned where the z-axis is the up axis. This orientation is crucial for maintaining consistency in spatial understanding and scene interpretation across different datasets and applications.
Example preprocessed point clouds, reconstructed from RGB videos using MASt3R-SLAM, are available in SpatialLM-Testset.
Use rerun to visualize the point cloud and the predicted structured 3D layout output:
bash
1# Convert the predicted layout to Rerun format2python visualize.py --point_cloud pcd/scene0000_00.ply --layout scene0000_00.txt --save scene0000_00.rrd
34# Visualize the point cloud and the predicted layout5rerun scene0000_00.rrd
Evaluation
To evaluate the performance of SpatialLM, we provide eval.py script that reports the benchmark results on the SpatialLM-Testset in the table below in section Benchmark Results.
1# Run inference on the PLY point clouds in folder SpatialLM-Testset/pcd with SpatialLM-Qwen-0.5B model2python inference.py --point_cloud SpatialLM-Testset/pcd --output SpatialLM-Testset/pred --model_path manycore-research/SpatialLM-Qwen-0.5B
34# Evaluate the predicted layouts5python eval.py --metadata SpatialLM-Testset/test.csv --gt_dir SpatialLM-Testset/layout --pred_dir SpatialLM-Testset/pred --label_mapping SpatialLM-Testset/benchmark_categories.tsv
SpatialLM Testset
We provide a test set of 107 preprocessed point clouds, reconstructed from RGB videos using MASt3R-SLAM. SpatialLM-Testset is quite challenging compared to prior clean RGBD scans datasets due to the noises and occlusions in the point clouds reconstructed from monocular RGB videos.
Benchmark results on the challenging SpatialLM-Testset are reported in the following table:
Method
SpatialLM-Llama-1B
SpatialLM-Qwen-0.5B
Floorplan
mean IoU
wall
78.62
74.81
Objects
F1 @.25 IoU (3D)
curtain
27.35
28.59
nightstand
57.47
54.39
chandelier
38.92
40.12
wardrobe
23.33
30.60
bed
95.24
93.75
sofa
65.50
66.15
chair
21.26
14.94
cabinet
8.47
8.44
dining table
54.26
56.10
plants
20.68
26.46
tv cabinet
33.33
10.26
coffee table
50.00
55.56
side table
7.60
2.17
air conditioner
20.00
13.04
dresser
46.67
23.53
Thin Objects
F1 @.25 IoU (2D)
painting
50.04
53.81
carpet
31.76
45.31
tv
67.31
52.29
door
50.35
42.15
window
45.4
45.9
License
SpatialLM-Llama-1B is derived from Llama3.2-1B-Instruct, which is licensed under the Llama3.2 license.
SpatialLM-Qwen-0.5B is derived from the Qwen-2.5 series, originally licensed under the Apache 2.0 License.
All models are built upon the SceneScript point cloud encoder, licensed under the CC-BY-NC-4.0 License. TorchSparse, utilized in this project, is licensed under the MIT License.
Citation
If you find this work useful, please consider citing:
bibtex
1@misc{spatiallm,
2 title = {SpatialLM: Large Language Model for Spatial Understanding},
3 author = {ManyCore Research Team},
4 howpublished = {\url{https://github.com/manycore-research/SpatialLM}},
5 year = {2025}
6}
Acknowledgements
We would like to thank the following projects that made this work possible:
