SparseVideoNav introduces video generation models to real-world beyond-the-view vision-language navigation for the first time. It pioneers a paradigm shift from continuous to sparse video generation for longer prediction horizons. By guiding trajectory inference with a generated sparse future spanning a 20-second horizon, it achieves sub-second inference (a 27× speed-up). It also marks the first realization of beyond-the-view navigation in challenging night scenes.
The model is designed for generating sparse future video frames based on a current visual observation (video) and a natural language instruction (e.g., "turn right"). It is primarily intended for research in Embodied AI, specifically Vision-Language Navigation (VLN) in real-world environments.
The model is a research prototype and is not intended for deployment in safety-critical real-world autonomous driving or robotic navigation systems without further extensive testing, safety validation, and fallback mechanisms.
Use the code below to get started with the model using our custom pipeline.
Ensure you have cloned the
GitHub repository and installed the requirements.
1from omegaconf import OmegaConf
2from inference import SVNPipeline
3
4# Load configuration
5cfg = OmegaConf.load("config/inference.yaml")
6cfg.ckpt_path = "/path/to/models/SparseVideoNav-Models" # Path to your downloaded weights
7cfg.inference.device = "cuda:0"
8
9# Initialize pipeline
10pipeline = SVNPipeline.from_pretrained(cfg)
11
12# Run inference (Returns np.ndarray (T, H, W, C) uint8)
13video = pipeline(video="/path/to/input.mp4", text="turn right")
1@article{zhang2026sparse,
2 title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation},
3 author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang},
4 journal={arXiv preprint arXiv:2602.05827},
5 year={2026}
6}
