Puffin-World is a unified multimodal world model that perceives, simulates, generates, and reconstructs the 3D world within one framework. Instead of representing a world only as a sequence of RGB frames, it explicitly models three complementary native 3D world states:
Physics: gravity fields and latitude maps that anchor observations and generated trajectories to the real world.
Geometry: dense depth that exposes the underlying 3D structure and supports direct reconstruction.
Appearance: high-fidelity images and sequences that remain spatially coherent as the camera moves.
A single integrated framework of a geometry-aligned vision encoder, an LLM, a diffusion model, and a lightweight connector supports physical-world perception, free-viewpoint spatial simulation, 3D world modeling, and closed-loop interaction without task-specific external perception or reconstruction modules.
At its core is camera-centric multi-view world modeling: given one initial view and a camera trajectory, Puffin-World jointly generates the remaining RGB and depth views and consolidates them into a gauge-aligned 3D point-cloud reconstruction.
Highlights
Native 3D world states: physics, geometry, and appearance are modeled together rather than leaving physical orientation and scene structure implicit.
Omni-Camera: a 9-channel dense camera condition combines a gravity-aware absolute perspective field with ray-based relative geometry, supporting camera-to-world grounding, rotation, translation, and compound motion.
Physics propagation: the gravity direction perceived from the reference view is propagated through the target camera trajectory to keep generated views in one coherent physical frame.
Unified understanding and generation: autoregressive camera understanding and diffusion-based generation share the same multimodal framework.
Scaled world modeling: Puffin-World is trained with Puffin-16M, comprising 15M vision-language-camera triplets and 1M diverse camera trajectories curated from 28 public datasets.
Results at a Glance
Capability
Result
Camera-to-world understanding
Best median errors in 12/12 comparisons and best AUC in 33/36 metrics, including ties, across Stanford2D3D, MegaDepth, TartanAir, and LaMAR
Camera-controllable generation
0.84° up-vector, 1.26° latitude, and 0.79° gravity median errors, with the lowest FID on Puffin-Cam-Bench
3D world modeling
PSNR (17.22) and LPIPS (0.318) on RealEstate10K
Challenging trajectories
Lowest median roll and pitch errors on Puffin-Traj-Bench: 0.80° and 1.10°
Puffin-World currently focuses on static scenes and models physics primarily through gravity and latitude. Dynamic environments, longer temporal horizons, richer object interactions, and broader physical states remain important directions for future work. Generated content may also inherit biases or failure modes from its training data and underlying pretrained components.
Citation
If the Puffin series supports your research, please consider citing our work:
bibtex
1@article{liao2026puffinworld,
2 title = {Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States},
3 author = {Liao, Kang and Luo, Yihang and Wu, Xiao-Ming and Jin, Linyi and Wu, Size and Lin, Chunyu and Zhao, Yao and Wang, Fei and Li, Wei and Loy, Chen Change},
4 journal = {arXiv preprint arXiv:2609.04196},
5 year = {2026}
6}
78@article{liao2025puffin,
9 title = {Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation},
10 author = {Liao, Kang and Wu, Size and Wu, Zhonghua and Jin, Linyi and Wang, Chao and Wang, Yikai and Wang, Fei and Li, Wei and Loy, Chen Change},
11 journal = {arXiv preprint arXiv:2510.08673},
12 year = {2025}
13}
