You can deploy this model and start earning money today!

Insta360-Research/
PanoWorld

Name: PanoWorld
Availability: InStock

2607.09661

Views

No views yet

PanoWorld logo
PanoWorld: Real-World Panoramic Generation

arXiv

Haoyuan Li¹ Dizhe Zhang¹✉ Yuemei Zhou¹ Xiangkai Zhang^1,2 Haoran Feng^1,3 Xiaofan Lin¹ Wenjie Jiang¹ Bo Du⁴ Ming-Hsuan Yang⁵ Lu Qi^1,4✉

¹Insta360 Research ²Institute of Automation Chinese Academy of Sciences ³Tsinghua University
⁴Wuhan University ⁵UC Merced

Quick try: demo assets in assets/demo/ — see Inference below.

Updates

[2026-07] 🎉 Initial release of code. Training code is coming soon.

Introduction

In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.

Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA). Then, a three-stage training pipeline is introduced to progressively optimize each component.

To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.

Environment Setup

Prerequisites

OS: Linux (tested on Ubuntu 22.04)
GPU: CUDA-compatible GPU with ≥20GB VRAM
CUDA: 12.8 or higher
Python: 3.10
FFmpeg: For video processing

Step 1: Create Conda Environment

Action Model inference assumes the PanoWorld conda env is already active.

bash

git clone https://github.com/Insta360-Research-Team/PanoWorld.git
cd PanoWorld

bash scripts/setup_panoworld_env.sh
conda activate PanoWorld

Or install manually:

bash

conda create -n PanoWorld python=3.10 -y
conda activate PanoWorld
pip install -e .
export PYTHONPATH="$(pwd)${PYTHONPATH:+:${PYTHONPATH}}"

Dependencies are listed in requirements.txt. After Step 1, continue with Step 2–3 (base model + PanoWorld checkpoints) before running inference_action.sh.

Step 2: Download Base Model (Wan2.2-TI2V-5B)

PanoWorld is built upon the Wan-AI Wan2.2-TI2V-5B video diffusion model.

bash

# Download using provided script
python scripts/download_wan2.2.py

# Or manually download from Hugging Face
# Visit: https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B
# Download to: models/Wan-AI/Wan2.2-TI2V-5B/

Step 3: Download Panoworld Models

Download the 480p or 720p checkpoints:

bash

# Option 1: Using our download script
python scripts/download_panoworld_models.py

# Option 2: Manual download from Hugging Face
# Visit: https://huggingface.co/Insta360-Research/PanoWorld

Final model directory structure:

models/
├── Wan-AI/
│   └── Wan2.2-TI2V-5B/
├── lora/
│   ├── 480p/480p_lora.safetensors
│   └── 720p/720p_lora.safetensors
├── action/
│   ├── 480p/480p_action.safetensors
│   └── 720p/720p_action.safetensors
└── casual/
    ├── dmd_chunkwise/model.pt
    └── rolling_dmd_480p_161/model.pt

Step 4: Install Causal-Forcing Package (Optional)

The Causal-Forcing stage requires additional dependencies:

Please refer to Causal-Forcing/README.md for installation instructions.

World360 Dataset

World360 comprises 120,000 high-quality sequences, unifying 70,000 curated real-world clips with 50,000 high-fidelity simulations from AirSim360, and introduces diverse multi-altitude aerial trajectories with precise camera poses and depth information.

Output: Each dataset generates:

81-frame & 161-frame panoramic video.
Camera trajectory csv file
PNG image sequence

Inference

Bundled demo assets live under assets/demo/. Resolution-specific cases are under 480/ and 720/; each case folder contains a 2:1 equirectangular panorama (input.jpg), text prompt (prompt.txt), and camera trajectory (pose.txt). 720p cases also include a reference video (reference_gen_joint_step2000.mp4).

Pose spacing: For custom pose.txt trajectories, we recommend a horizontal step of ~0.05 between consecutive frames, i.e. sqrt(action_dx² + action_dy²) ≈ 0.05 in the model world frame (-x forward, +y left). The bundled demo poses use this spacing.

Camera-Controlled Video Generation

High-quality panoramic I2V with camera trajectory control.
Unified entry: inference_action.sh

Output: {output}/gen_video.mp4 (single sample) or gen_video_{i}.mp4 (batch).

bash

# --- Demo: image + prompt + synthetic forward motion ---
./inference_action.sh \
  --resolution 480 \
  --image assets/demo/input.jpg \
  --prompt "$(cat assets/demo/prompt.txt)" \
  --motion forward \
  --output ./inference_output/demo_forward

# --- Demo case with recorded pose (480p) ---
CASE=assets/demo/480/case2_waterway_slice000
./inference_action.sh \
  --resolution 480 \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_action

# --- Demo case with recorded pose (720p) ---
CASE=assets/demo/720/case1_waterway_slice706
./inference_action.sh \
  --resolution 720 \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_action

# 480p — single sample from test CSV
./inference_action.sh \
  --resolution 480 \
  --prompt_nums 1 \
  --output ./inference_output/demo_480p

# 720p — native 1408×704, upscaled to 1440×720
./inference_action.sh \
  --resolution 720 \
  --prompt_nums 1 \
  --output ./inference_output/demo_720p

# Or via top-level wrapper
RESOLUTION=480 PROMPT_NUMS=1 ./inference_preview.sh
RESOLUTION=720 PROMPT_NUMS=1 ./inference_preview.sh

python Action-Model/infer_action.py --help

Flag	Description
`--resolution`	`480` (960×480) or `720` (1408×704 → 1440×720)
`--image` / `--prompt`	Demo mode: panoramic image + text prompt (2:1 image)
`--motion`	`forward` / `backward` / `left` / `right` / `up` /`down` /, or a pose file (demo mode)
`--output`	Output directory
`--output_filename`	Default `gen_video.mp4`
`--prompt_path`	CSV with columns `video`, `short_prompt`, `pose_path` (batch mode)
`--prompt_nums`	Number of CSV rows to run (batch mode)

Default test CSVs: data_test.csv (480p), data_test_720p.csv (720p).

Causal Forcing Stage

Real-time panoramic generation with Causal Forcing on Wan2.2 5B.
Entry: Casual-forcing/inference_causal.sh (requires causal_forcing env).

Output: {output}/causal_video.mp4

bash

cd Casual-forcing

# --- Quick demo (root assets, 480p case1) ---
./inference_causal.sh \
  --image ../assets/demo/input.jpg \
  --prompt "$(cat ../assets/demo/prompt.txt)" \
  --output ../assets/demo/out_short

# --- Demo case with recorded pose ---
CASE=../assets/demo/480/case2_waterway_slice000
./inference_causal.sh \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_pose

# --- 161-frame long video (Rolling Forcing) ---
CASE=../assets/demo/480/case1_park_slice002
./inference_causal.sh \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --frames 161 \
  --output ${CASE}/out_long

# --- Synthetic motion presets ---
./inference_causal.sh \
  --image ../assets/demo/input.jpg \
  --prompt "$(cat ../assets/demo/prompt.txt)" \
  --motion forward \
  --output ../assets/demo/out_forward

python infer_causal.py --help

Flag	Description
`--frames`	`81` (short, default) or `161` (long / Rolling Forcing)
`--motion`	`forward` / `backward` / `left` / `right`, or a pose file (`.txt` / `.csv` / `.npy`)
`--output`	Output directory; video is always `causal_video.mp4`

Memory Preview

Coming soon. Long-range memory-augmented 720p inference (161-frame) will be released in a future update.

Training

PanoWorld training is organized in stages:

Stage 1  Video LoRA     →  output/lora_480p | lora_720p
Stage 2  Action Model   →  output/action_480p | action_720p  (requires Stage 1)
Stage 3  Memory         →  coming soon
Stage 4  Causal Forcing →  Casual-forcing/logs/  (480p, see below)

Use RESOLUTION=480|720 to switch presets (Matrix-3D convention). Config: configs/resolution.yaml.

Training CSVs must include panoramic video paths and text prompts. Override defaults with DATA_CSV=/path/to/your.csv.

Resolution	Native train	Output	LoRA rank
480	960×480	960×480	64
720	1408×704	1440×720 (inference upscale)	256

Stage 1 — Video LoRA

Fine-tune Wan2.2 I2V with LoRA for panoramic video generation (81 frames, no camera control module).

bash

# 480p
RESOLUTION=480 bash scripts/train/01_video_lora.sh

# 720p (recommended for Action Model 720p)
RESOLUTION=720 bash scripts/train/01_video_lora.sh

Checkpoints are saved under {OUTPUT}/checkpoints/ with latest.json for resume.

Legacy wrappers: scripts/train/01_video_lora_480p.sh, 01_video_lora_720p.sh.

Stage 2 — Action Model

Freeze LoRA and train the cam_self_attn camera-control module (trajectory-conditioned I2V). Requires Stage 1 LoRA.

bash

# After Stage 1 completes
RESOLUTION=480 bash scripts/train/02_action.sh
RESOLUTION=720 bash scripts/train/02_action.sh

# Warm-start from a simulation-pretrained action checkpoint (optional)
INIT_ACTION=/path/to/simulation_action.safetensors \
  RESOLUTION=720 bash scripts/train/02_action.sh

Checkpoints: {OUTPUT}/checkpoints/. Use the latest action weights with inference_action.sh for preview inference.

720p project-specific script (hardcoded paths): Action-Model/train_720p.sh.

Stage 3 — Memory

Coming soon.

Stage 4 — Causal Forcing

See Casual-forcing/README.md for installation and training.

Checkpoint Layout

output/
├── lora_480p/checkpoints/
├── lora_720p/checkpoints/
├── action_480p/checkpoints/
└── action_720p/checkpoints/
Casual-forcing/logs/          # Causal Forcing training outputs

Project Structure

PanoWorld/
├── assets/demo/                # Demo inputs (480/720 cases: input, prompt, pose)
│   ├── 480/                    # 960×480 cases
│   └── 720/                    # 1408×704 cases (+ reference_gen_*.mp4)
├── inference_action.sh         # Action Model unified entry → gen_video.mp4
├── Action-Model/               # Action Model Python scripts (480p / 720p)
│   └── infer_action.py         # unified CLI
├── Casual-forcing/             # Causal Forcing inference & training
│   ├── infer_causal.py         # unified CLI → causal_video.mp4
│   └── inference_causal.sh
├── diffsynth/                  # Shared DiffSynth modules
├── scripts/train/              # Staged training launchers (01–04)
├── models/                     # Wan2.2 base + PanoWorld checkpoints
│   ├── lora/                   # Video LoRA (480p / 720p)
│   ├── action/                 # Action Model (480p / 720p)
│   └── casual/                 # Causal Forcing inference weights
├── inference_preview.sh        # RESOLUTION=480|720 wrapper
├── utils/
│   ├── run_metrics.sh        # Unified 480p/720p quality evaluation
│   ├── resolution_config.py    # Resolution presets helper
│   └── validate/             # step1_validate, step2_tractory, metric modules
└── configs/resolution.yaml     # 480 / 720 presets

Tools

Panoramic video quality evaluation — unified script for 480p / 720p: utils/run_metrics.sh (implementation: utils/validate/).

Metrics: PSNR (windowed), FID / FID_pole / FID_equ, FAED, NIQE, QAlign (Quality / Aesthetic).

bash

RESOLUTION=480 INFERENCE_DIR=/path/to/inference RESULT_DIR=/path/to/result bash utils/run_metrics.sh
RESOLUTION=720 INFERENCE_DIR=/path/to/inference RESULT_DIR=/path/to/result bash utils/run_metrics.sh

Results are written to RESULT_DIR (PSNR CSV + eval_results/eval_results_*.txt).

Acknowledgments

We thank the following projects for their inspiring work, our code is partially based on the code from these projects:

Wan-AI: Base video diffusion model
UCPE: Camera-controlled text-to-video generation.
Causal-Forcing: Causal-Forcing distillation for fast diffusion models

Citation

If you find Panoworld useful for your research, please cite:

bibtex

@misc{li2026panoworldrealworldpanoramicgeneration,
      title={PanoWorld: Real-World Panoramic Generation}, 
      author={Haoyuan Li and Dizhe Zhang and Yuemei Zhou and Xiangkai Zhang and Haoran Feng and Xiaofan Lin and Wenjie Jiang and Bo Du and Ming-Hsuan Yang and Lu Qi},
      year={2026},
      eprint={2607.09661},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.09661}, 
}

Insta360-Research/PanoWorld

PanoWorld logo PanoWorld: Real-World Panoramic Generation

Updates

Introduction

Environment Setup

Prerequisites

Step 1: Create Conda Environment

Step 2: Download Base Model (Wan2.2-TI2V-5B)

Step 3: Download Panoworld Models

Step 4: Install Causal-Forcing Package (Optional)

World360 Dataset

Inference

Camera-Controlled Video Generation

Causal Forcing Stage

Memory Preview

Training

Stage 1 — Video LoRA

Stage 2 — Action Model

Stage 3 — Memory

Stage 4 — Causal Forcing

Checkpoint Layout

Project Structure

Tools

Acknowledgments

Citation

Insta360-Research/
PanoWorld

PanoWorld logo
PanoWorld: Real-World Panoramic Generation