On-device LiDAR scene capture → URDF / MJCF / PLY, for iPhone.
Twin3D detecting a keyboard with an oriented 3D bounding box
This repository hosts the exported CoreML model packages that ship inside the Twin3D iOS app. They are format-conversions of Meta Boxer and Ultralytics YOLOE — no retraining, no weight modification — and inherit their licensing (CC-BY-NC 4.0 and AGPL-3.0 respectively).
Use them to avoid the Python export step when building the iOS app or when integrating with a non-iPhone capture stack via CoreML.
LiDAR point cloud with oriented bounding box on the left, MuJoCo simulation on the right
fp32 + MLComputeUnits.cpuAndGPU is the default configuration, and the one with the tightest parity against the PyTorch reference. See the parity section in the main repo for the measured bounds and the caveats around fp16 on the Neural Engine.
Using the models
In the Twin3D iOS app
Clone the Twin3D repo, then copy the exports directly into the app bundle:
For RealSense, ZED, OAK-D, Azure Kinect, or ROS bags, use the ONNX export path from the main repo — CoreML packages are iPhone-specific. See Adapting to other sensors.
One tap captures one frame of RGB + LiDAR depth + camera intrinsics + gravity, runs YOLOE for 2D detection, lifts each box with BoxerNet, and writes the four scene files. All on-device.
Colored LiDAR point cloud with an oriented bounding box around a detected keyboard
Model I/O summary
YOLOE (2D detection)
Input: 640 × 640 RGB, [0, 1] float
Output: N × (class, xmin, ymin, xmax, ymax, score)
Open vocabulary, frozen at export time (see VOCAB in the main repo's tools/export_yoloe.py)
BoxerNet (3D lifting)
Inputs: 960 × 960 RGB, 60 × 60 depth patches (median per 16×16 tile), N × 6 Plücker rays per patch, fx fy cx cy intrinsics, gravity 3-vector, 2D boxes
Outputs per box: center ∈ ℝ³, size ∈ ℝ³, yaw, confidence
DINOv3 ViT-S/16+ backbone is inlined — the .pth is not needed at inference time.
License and attribution
CC-BY-NC 4.0. Non-commercial research use only.
This repository redistributes CoreML format-conversions of:
Meta Boxer (CC-BY-NC 4.0) — the 3D lifter and the DINOv3 backbone. See facebook/boxer for the original PyTorch checkpoints. Cite the Boxer paper (below) if you use the model in research.
Ultralytics YOLOE (AGPL-3.0) — the 2D detector. Any network-served derivative inherits AGPL copyleft obligations; Ultralytics offers a commercial license for users who cannot comply.
No weight modifications have been made beyond format conversion (PyTorch → CoreML) and, for the fp16 variant, precision cast.
For a commercial deployment, replace both the 3D lifter and the 2D detector with permissively licensed equivalents — see the Roadmap in the main repo.
1@article{boxer2026,
2 title = {Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D},
3 author = {DeTone, Daniel and Shen, Tianwei and Zhang, Fan and Ma, Lingni
4 and Straub, Julian and Newcombe, Richard and Engel, Jakob},
5 year = {2026},
6 url = {https://arxiv.org/abs/2604.05212}
7}
