EVS — Extrapolative Visual Sensing with Geometry-Aware Sensor Tokens
A generative video diffusion model for posed scene extrapolation: given a short observed video
and a target camera trajectory, generate the video along that trajectory — extrapolating into unobserved
regions while staying consistent on revisit (when the camera returns to previously-seen content).
Built on Wan2.1-VACE-14B, trained with Diffusion Forcing, and conditioned via Plücker camera
control + a geometry-aware "sensor token" design.
Method
Diffusion Forcing (in-context memory). Independent per-frame noise levels; the observed frames sit in
the same latent sequence at noise ≈ 0 (clean) while target frames are denoised. Memory is in-context — the
model's self-attention reads the clean observed frames; no separate memory module.
Sensor token (the contribution). Each token carries, beyond its content latent:
Ray position embedding — the token's world-frame Plücker ray, added to self-attention Q/K, so tokens
that view the same content (e.g. a revisited camera pose) retrieve each other by geometry.
Evidence weight — a depth-free depth-hypothesis co-visibility count (how many frames observe the
token's content), injected as an attention key-boost + a content channel (and a knownness prior).
Camera control. PermaVid-style Plücker SimpleAdapter, additive at the patch embedding (not via the
VACE control branch). Per-frame timesteps require no architecture surgery (Wan DiTBlock already accepts
per-token modulation). All new modules are zero-init, so an untrained model equals the base Wan T2V model.
Data
RealEstate10K (static real-estate walkthroughs, smooth cinematic camera). Clips are built as smooth
out-and-back trajectories over consecutive frames (forward then retrace) for a genuine revisit with real
GT, plus a forward-split eval mode (observe a prefix, extrapolate the forward continuation into unseen).
Geometry (ray codes + evidence) is a pure function of the poses and is precomputed/cached.
Work in progress. Model + pipeline implemented and verified; training on RealEstate10K. Checkpoints and
qualitative samples: see the companion HuggingFace repo.
