Déjà View: Looping Transformers for Multi-View 3D Reconstruction
Alessandro Burzio*,
Tobias Fischer*,
Sven Elflein,
Qunjie Zhou,
Riccardo de Lutio,
Jiawei Ren,
Jiahui Huang,
Shengyu Huang,
Marc Pollefeys,
Laura Leal-Taixé,
Zan Gojcic+,
Haithem Turki+
Model Overview
Description
Déjà View Looping Transformer (DVLT) is a feed-forward three-dimensional (3D) reconstruction model that takes unposed Red, Green, Blue (RGB) images or video and predicts per-pixel depth, ray maps (and thus 3D points), and per-view camera intrinsics/extrinsics in a single pass.
Novelty: A weight-tied looped transformer — instead of stacking many distinct layers, a single shared block is applied for K refinement steps over a DINOv2-initialized per-view state, with each step conditioned on a continuous time interval (t_k, t_k+1) ⊂ [0, 1]. A single checkpoint exposes the iteration count K as an inference-time compute/quality knob without retraining separate models (released checkpoint valid for K ∈ [8, 16]).
This model is for research and development only.
License/Terms of Use
Model. The model (checkpoints, learned weights, and configuration files) is released under the
NVIDIA License:
https://huggingface.co/nvidia/dvlt/blob/main/LICENSE.txt.
Source code. The accompanying source code is licensed separately. The repository is primarily licensed under the Apache License, Version 2.0 — see the LICENSE file at the repository root. Portions of the codebase derived from VGGT (Meta) are distributed under the VGGT License v1; the full text is provided in LICENSES/VGGT-LICENSE.txt. Full third-party attribution, per-file notices, and upstream license texts are collected in THIRD_PARTY_LICENSES.md.
Deployment Geography
Global
Use Case
Primary users:
- Computer Vision Researchers: For benchmarking multi-view 3D reconstruction, studying weight-tied / recurrent transformer architectures, and developing neural rendering pipelines.
- Augmented Reality/Virtual Reality & Robotics Engineers: For real-time simultaneous localization and mapping (SLAM), scene understanding, and navigation research prototypes.
- 3D Content Creators: For rapid conversion of unposed video/image collections into 3D assets.
Primary Use Cases:
- 3D Reconstruction: Fast, feed-forward estimation of dense per-pixel depth, ray maps, and per-view camera poses from unposed images or video, without iterative optimization.
- Structure-from-motion (SfM) Replacement: Accelerating initialization for 3D Gaussian Splatting and Neural Radiance Field (NeRF) training by replacing slow SfM pipelines (e.g., COLMAP).
- Compute-Adaptive Inference: A single checkpoint supports a range of recurrence step counts (8–16) at inference, letting downstream applications trade reconstruction quality for latency without retraining.
Release Date
GitHub 06/02/2026 via [URL TBD]
Hugging Face (HF) 06/02/2026 via
https://huggingface.co/nvidia/dvlt References
Quick Start
Install the
code (preferably in a conda environment):
1conda create -n dvlt python=3.12 && conda activate dvlt
2conda install pytorch=2.5.1 torchvision pytorch-cuda=12.4 -c pytorch -c nvidia -c conda-forge
3pip install -e .[all]
Run feed-forward 3D reconstruction on a directory of images, a video, or an explicit list of frames:
1import torch
2from accelerate import Accelerator
3
4from dvlt.model.dvlt.model import DVLT
5from dvlt.util.preprocess import load_sequence, preprocess_images
6
7accelerator = Accelerator(mixed_precision="bf16")
8
9model = DVLT(img_size=504)
10model.load_pretrained("nvidia/dvlt") # local dir, HTTPS URL, or HF Hub repo id
11model.setup_test(accelerator)
12
13# load_sequence accepts a directory, a single video, or an explicit list of files.
14_, frames = load_sequence("path/to/scene_dir")
15batch = preprocess_images(frames, img_size=504, patch_size=14, device=accelerator.device)
16
17with torch.no_grad(), accelerator.autocast():
18 predictions = model.predict(batch, accelerator)
19
20cameras = predictions["cameras"][0] # Cameras object with shape [S]
21extrinsics_c2w = cameras.camera_to_worlds # (S, 3, 4) — OpenCV convention [R | t]
22intrinsics = cameras.get_intrinsics_matrices() # (S, 3, 3)
23depths = predictions["depths"][0] # (S, H, W)
24world_points = predictions["world_points"][0] # (S, H, W, 3)
Model Architecture
Architecture Type: Transformer
Network Architecture: Vision Transformer (ViT) with a single weight-tied looped block applied recurrently for K refinement steps.
Backbone: DINOv2 ViT-B (patch size 14).
Number of model parameters: 1.17 * 10^8
Input(s)
Input Type(s): Image, video
Input Format(s):
- Image: Red, Green, Blue (RGB)
- Video: .mov, .mp4 (decoded to frames)
Input Parameters:
- Image collection: Three-Dimensional (3D) — a sequence/stack of two-dimensional (2D) RGB images per scene.
- Video: Three-Dimensional (3D) — a sequence of two-dimensional (2D) frames per clip.
Other Properties Related to Input:
- Video is decoded to frames at the video's native frame rate.
- Training resolution: 504-pixel longest edge; sides padded to multiples of patch size 14.
- Number of views per scene at training time: V ∈ [2, 18]; inference supports a wider range (memory-bound).
Output(s)
Output Type(s):
- Per-pixel depth map
- Per-pixel ray map (origin + unnormalized direction)
- Per-view camera parameters (extrinsics, intrinsics)
Output Format(s):
- Depth map: scalar metric distance per pixel
- Ray map: 3D origin + 3D direction per pixel (6 channels)
- Point cloud (derived analytically as X = R^o + D · R^d, i.e. ray origin plus depth times ray direction): X, Y, Z per pixel
- Camera intrinsics: focal length, principal point
- Camera extrinsics: rotation matrix (camera-to-world), translation vector
Output Parameters:
- Depth / confidence: Two-Dimensional (2D) per view (H × W)
- Ray map / point cloud: Three-Dimensional (3D) per view (H × W × 3)
- Camera intrinsics: Two-Dimensional (2D) (3 × 3)
- Camera extrinsics: Two-Dimensional (2D) (4 × 4)
Other Properties Related to Output:
- One 3D point, one depth value, one ray, and one depth-confidence value are predicted per input pixel in each image.
- One set of camera parameters (intrinsics + extrinsics) is predicted per input image, expressed in the coordinate frame of the first view.
- Cameras are recovered from the predicted ray maps following Lin et al. 2025; a camera multi-layer perceptron (MLP) head provides a faster auxiliary alternative.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., Compute Unified Device Architecture (CUDA) libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Software Integration
Runtime Engine(s): PyTorch 2.5 (Hugging Face Accelerate)
Supported Hardware Microarchitecture Compatibility:
- NVIDIA Ampere
- NVIDIA Blackwell
- NVIDIA Hopper
- NVIDIA Lovelace
Preferred/Supported Operating System(s): Linux
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s)
- v1.0: Initial release. DINOv2 ViT-B backbone (patch size 14), two-stage training (200,000 iterations stage 1 + 40,000 iterations stage-2 depth-decoder finetune) on 128 H100 GPUs, K ~ Beta(2, 1) on [8, 16].
Training, Testing, and Evaluation Datasets
DVLT is trained on a mixture of 28 public datasets. Per-dataset realized mix shares (per-epoch frame-view sampling under temperature α = 0.5 on per-dataset frame counts) are listed inline below.
Training Datasets
Aria Synthetic Environments
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 13.77%
Properties: Large-scale synthetic dataset of egocentric video sequences in realistic indoor environments, rendered with associated ground-truth camera trajectories, depth, semantic / instance segmentation, and 3D scene layout.
Wild RGB-D
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Automatic/Sensors]
Mix share: 9.32%
Properties: Large-scale dataset of ~20,000 RGB + Depth (RGB-D) videos of objects captured in the wild using iPhones. Includes camera poses and object masks.
TRELLIS
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 9.04%
Properties: Multi-view renderings of 3D objects produced via the TRELLIS rendering pipeline (Microsoft). We use 5 of the 6 TRELLIS subsets — 3D-FUTURE, ABO (Amazon Berkeley Objects), HSSD (Habitat Synthetic Scenes Dataset), Toys4K, and the Objaverse-XL Sketchfab subset — and explicitly exclude the Objaverse-XL GitHub subset. Each upstream asset collection retains its own license.
Common Objects in 3D (CO3D)
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Automatic/Sensors]
Mix share: 8.84%
Properties: ~6 million images. Large-scale dataset of object-centric videos captured with smartphones, annotated with camera poses and point clouds via structure-from-motion.
Taskonomy
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Automatic/Sensors]
Labeling Method by dataset [Automatic/Sensors]
Mix share: 7.26%
Properties: ~4 million indoor images across 600 buildings, captured with structured-light depth sensors. Provides per-pixel depth, surface normals, semantic / instance segmentation, and camera poses. Loaded via the
OmniData multi-view preprocessing pipeline.
ScanNet
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Hybrid: Automatic/Sensors, Human]
Mix share: 6.23%
Properties: RGB-D video dataset containing 2.5 million views in more than 1,500 scans, annotated with 3D camera poses, surface reconstructions, and instance-level semantic segmentations. Captured with an iPad-mounted depth sensor and crowdsourced semantic annotation.
DL3DV-10K
Data Modality: Image, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Automatic/Sensors]
Mix share: 5.84%
Properties: Large-scale scene dataset containing over 10,000 videos of diverse real-world scenes captured for deep-learning 3D vision. Camera poses are obtained using structure-from-motion.
Cubify Anything
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Automatic/Sensors]
Labeling Method by dataset [Hybrid: Human, Automatic/Sensors]
Mix share: 5.73%
Properties: Aligns handheld RGB-D captures from ARKitScenes with laser scans, creating a large-scale labeled dataset for 3D object detection.
TartanAir V2
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 4.78%
Properties: Updated, photorealistic synthetic SLAM dataset with diverse environments, motion patterns, and lighting/weather conditions. Provides RGB, depth, and camera trajectories.
Parallel Domain 4D
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 4.75%
Properties: Synthetic video dataset rendered using the
Parallel Domain synthetic data engine, designed for dynamic novel-view synthesis and 4D reconstruction. Dataset assembled and released by GCD (Van Hoorick et al., ECCV 2024).
ScanNet++
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Hybrid: Human, Automatic/Sensors]
Mix share: 3.96%
Properties: Millions of frames across ~460 scenes. High-resolution real-world 3D indoor semantic scene understanding dataset using laser scanning and DSLR/iPhone capture. The evaluation set uses a clean scene-level split disjoint from the training scenes.
Mid-Air
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 2.61%
Properties: ~420,000 video frames. Synthetic dataset for low-altitude drone flight, featuring multi-modal data (RGB, depth, normals, stereo) across different climate conditions.
Dynamic Replica
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 2.16%
Properties: 145,200 stereo frames (from 524 videos). Synthetic rendered videos of scanned environments with dynamic objects (people / animals); includes stereo images, disparity maps, and camera parameters.
TartanAir
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 2.13%
Properties: Challenging visual SLAM dataset collected in photorealistic simulation environments with various lighting / weather conditions.
BEDLAM
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 1.99%
Properties: Synthetic dataset of moving human bodies in 3D environments (~10K sequences), with photorealistic rendering and ground-truth depth / camera parameters.
BlendedMVG
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 1.35%
Properties: Higher-resolution / extended-scenes variant from the BlendedMVS family. 3D models are reconstructed from real images, then rendered to create synthetic training pairs blended with the original images.
MegaDepth
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Automated]
Labeling Method by dataset [Automatic/Sensors]
Mix share: 1.16%
Properties: ~150,000 reconstructed images derived from internet photo collections (Flickr) using structure-from-motion (SfM) and multi-view stereo (MVS) to generate depth maps.
Replica
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Automatic/Sensors]
Labeling Method by dataset [Automatic/Sensors]
Mix share: 0.95%
Properties: 18 high-fidelity reconstructions of real-world indoor spaces (apartments, hotel rooms, offices) captured with a custom RGB-D capture rig and dense surface reconstructions. Loaded via the
OmniData multi-view preprocessing pipeline.
Virtual KITTI 2
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 0.83%
Properties: Photorealistic synthetic video dataset mimicking the real KITTI dataset, used for object detection, tracking, and depth estimation.
Hypersim
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 0.69%
Properties: ~77,400 images. Photorealistic synthetic dataset for indoor scene understanding. Contains RGB, depth, semantic segmentation, and camera parameters.
Kubric
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 0.60%
Properties: A data-generation pipeline using PyBullet and Blender to create photo-realistic synthetic videos with rich annotations. We use the dataset built from assets from
Google Scanned Objects and environments from
HDRI Haven.
GTA-SfM
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 0.52%
Properties: Photorealistic synthetic dataset derived from GTA-V environments, providing ground-truth depth, camera poses, and dense SfM-style supervision in urban scenes.
MatrixCity
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 0.49%
Properties: Large-scale synthetic city dataset for neural rendering, generated using Unreal Engine 5. Both the aerial small-city-block subset and the street-level subset are used.
Mapillary Planet-Scale Depth dataset
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Hybrid: Human, Automatic/Sensors]
Labeling Method by dataset [Automatic/Sensors]
Mix share: 0.49%
Version: v1.1
Properties: ~750,000 images. Dense depth dataset derived from street-level imagery using SfM, covering a wide variety of locations and camera models.
Mapillary Metropolis
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Automatic/Sensors]
Labeling Method by dataset [Hybrid: Human, Automatic/Sensors]
Mix share: 0.47%
Version: v1
Properties: ~28,000 images. City-scale dataset containing street-level panoramic images, LiDAR point clouds, and aerial imagery.
UnrealStereo4K
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 0.44%
Properties: High-resolution synthetic stereo dataset generated using Unreal Engine 4.
MVS-Synth
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 0.44%
Properties: Photorealistic synthetic urban dataset, rendered from GTA-V, with ground-truth depth and camera poses for multi-view stereo.
Spring Benchmark
Data Modality: Image, Depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Synthetic]
Labeling Method by dataset [Synthetic]
Mix share: 0.40%
Properties: High-resolution photo-realistic computer-generated imagery (Blender) for optical flow and stereo benchmarking.
Testing / Evaluation Datasets
DVLT is evaluated (and tested) on five public reconstruction benchmarks spanning indoor, outdoor, object-centric, and driving scenes.
DTU
Data Modality: Image, depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Automatic/Sensors]
Properties: An industrial robot arm was mounted with a structured-light scanner. This allowed for structured-light scans corresponding to each image in the dataset. Images were taken by one of the cameras in the structured-light scanner.
ETH3D Stereo Benchmark
Data Modality: Image, depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Automatic/Sensors]
Properties: The multi-view stereo / 3D reconstruction benchmark covers a variety of indoor and outdoor scenes. Ground-truth geometry has been obtained using a high-precision laser scanner. A DSLR camera and a synchronized multi-camera rig with varying field of view were used to capture images.
7-Scenes
Data Modality: Image, depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Automatic/Sensors]
Properties: 7-Scenes is an RGB-D dataset featuring seven indoor scenes captured with a Kinect camera. The dataset includes RGB images (640 × 480), depth maps, and camera-to-world poses.
ScanNet++
Data Modality: Image, depth, Camera extrinsics and intrinsics.
Image Training Data Size [1 Million to 1 Billion Images]
Data Collection Method by dataset [Human]
Labeling Method by dataset [Hybrid: Human, Automatic/Sensors]
Properties: High-resolution real-world 3D indoor semantic scene understanding dataset using laser scanning and DSLR / iPhone capture. ScanNet++ also serves as a training dataset for multiple baselines and for DVLT; we enforce a clean scene-level split between the training and evaluation data.
nuScenes
Data Modality: Image, depth, Camera extrinsics and intrinsics.
Image Training Data Size [Less than a Million Images]
Data Collection Method by dataset [Hybrid: Human, Automatic/Sensors]
Labeling Method by dataset [Hybrid: Human, Automatic/Sensors]
Properties: Autonomous-driving dataset of 1,000 scenes collected in Boston and Singapore with a full sensor suite (6 cameras, 1 LiDAR, 5 radars, GPS, IMU). Provides synchronized multi-view imagery, LiDAR-derived depth, and ego-poses, used here as the driving-domain evaluation benchmark.
Inference
Acceleration Engine: PyTorch native (bf16 via Hugging Face Accelerate)
Test Hardware: NVIDIA H100, NVIDIA A100
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns
here.
Citation
If you find this work useful, please cite:
1@article{burzio2026dejaview,
2 title = {D\'ej\`a View: Looping Transformers for Multi-View 3D Reconstruction},
3 author = {Burzio, Alessandro and Fischer, Tobias and Elflein, Sven and Zhou, Qunjie and de Lutio, Riccardo and Ren, Jiawei and Huang, Jiahui and Huang, Shengyu and Pollefeys, Marc and Leal-Taix{\'e}, Laura and Gojcic, Zan and Turki, Haithem},
4 journal = {arXiv preprint arXiv:2605.30215},
5 year = {2026}
6}
