WildDet3D: Scaling Promptable 3D Detection in the Wild
WildDet3D is a promptable monocular 3D object detection model that detects and localizes objects in 3D from a single RGB image. It supports text, box, and point prompts for open-vocabulary 3D detection across diverse in-the-wild scenes.
Authors: Weikai Huang, Jieyu Zhang, Sijun Li, Taoyang Jia, Jiafei Duan, Yunqian Cheng, Jaemin Cho, Matthew Wallingford, Rustin Soraki, Chris Dongjoo Kim, Shuo Liu, Donovan Clay, Taira Anderson, Winson Han, Ali Farhadi, Bharath Hariharan, Zhongzheng Ren, Ranjay Krishna
Affiliations: Allen Institute for AI (Ai2), University of Washington, Cornell University, UNC-Chapel Hill
Model Details
| Property | Value |
|---|
| Backbone | SAM3 ViT (1024-dim, 32 blocks, patch 14) |
| Depth Backend | LingBot-Depth (DINOv2 ViT-L/14) |
| Parameters | ~1.2B |
| Input | RGB image + camera intrinsics (optional) + sparse/dense depth (optional) |
| Output | 2D boxes, 3D boxes, depth maps, predicted intrinsics |
| Prompt Types | Text, Box (visual/geometric), Point |
| License | SAM License |
When camera intrinsics are not available (e.g., in-the-wild images), the model can predict intrinsics internally. When sparse or dense depth (e.g., from LiDAR) is provided, it is fused for improved 3D localization.
Citation
1@article{wilddet3d,
2 title={WildDet3D: Scaling Promptable 3D Detection in the Wild},
3 author={Huang, Weikai and Zhang, Jieyu and Li, Sijun and Jia, Taoyang and Duan, Jiafei and Cheng, Yunqian and Cho, Jaemin and Wallingford, Matthew and Soraki, Rustin and Kim, Chris Dongjoo and Liu, Shuo and Clay, Donovan and Anderson, Taira and Han, Winson and Farhadi, Ali and Hariharan, Bharath and Ren, Zhongzheng and Krishna, Ranjay},
4 year={2026},
5}
License
This model uses SAM 3 and LingBot-Depth weights, and is licensed under the
SAM License. This model is intended for research and educational use in accordance with Ai2's
Responsible Use Guidelines.
Paper
