You can deploy this model and start earning money today!

Orange-3DV-Team/
MoVerse

Name: MoVerse
Availability: InStock

panorama

3d-gaussian-splatting

Views

No views yet

MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold

Overview

MoVerse builds a navigable 3D world from a single narrow-field-of-view image and renders interactive video in real time on a single RTX 4090.

Pipeline:

Panoramic Generation — Expands a single NFOV image into a 360° ERP panorama via topology-aware latent diffusion
3D Gaussian Scaffold — Lifts the panorama into a panoramic 3D Gaussian scaffold using feed-forward residual prediction
Autoregressive Video Refinement — Translates scaffold renderings along user camera trajectories into photorealistic video at 8 FPS

Model Files

File	Path	Description
AutoLevel	`gimbal360/autolevel.pth`	Camera calibration model for panorama alignment
Pano LoRA	`gimbal360/pytorch_lora_weights.safetensors`	FLUX.1-Fill-dev LoRA adapter for topology-aware panorama outpainting
Gaussian Predictor	`gaussian_predictor_checkpoint.pth`	Feed-forward 3DGS predictor (includes DINOv3 encoder, no separate DA2 needed)
Self-Forcing Video	`self_forcing_video.pt`	Autoregressive video generation checkpoint (bf16, Self-Forcing on Wan2.1-T2V-1.3B)
TAEHV VAE	`taew2_1.pth`	Lightweight VAE decoder for real-time frame decoding

Usage

Download all weights

bash

# From the MoVerse GitHub repo
./scripts/download_weights.sh

Or download individual files:

bash

# Install huggingface-cli
pip install huggingface_hub

# Download all MoVerse weights to checkpoints/
huggingface-cli download Orange-3DV-Team/MoVerse --local-dir checkpoints

# Or download a single file
huggingface-cli download Orange-3DV-Team/MoVerse self_forcing_video.pt --local-dir checkpoints

Additional open-source models (not in this repo)

The full pipeline also requires these open-source models, downloaded automatically by download_weights.sh:

Model	HuggingFace Repo	Path
FLUX.1-Fill-dev	`black-forest-labs/FLUX.1-Fill-dev`	`checkpoints/FLUX.1-Fill-dev/`
Depth Anything 3	`depth-anything/DA3NESTED-GIANT-LARGE-1.1`	`checkpoints/DA3NESTED-GIANT-LARGE-1.1/`
Wan2.1-T2V-1.3B	`Wan-AI/Wan2.1-T2V-1.3B`	`checkpoints/Wan2.1-T2V-1.3B/`

Full Setup & Quick Start

See the GitHub repository for complete installation, build, and usage instructions.

Minimum requirements: NVIDIA GPU with 24 GB VRAM, Linux, 64 GB RAM, Python 3.10.

Citation

bibtex

@article{moverse2026,
  title   = {MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold},
  author  = {Yang Zhou and Ziheng Wang and Yuqin Lu and Haofeng Liu and Jun Liang and Shengfeng He and Jing Li},
  journal = {arXiv preprint arXiv:2606.13376},
  year    = {2026}
}

Acknowledgements

This work builds on Self-Forcing, Wan2.1, gsplat, FLUX.1-Fill-dev, Depth Anything 3, Apple ML-Sharp, GEN3C, CausVid, RAVEN, and MemRoPE.

Project Page | Paper | GitHub
_{Built by Orange Team, Youku Moku-Lab}