Official Code for the paper "Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics" accepted at ECCV 2026.
💡 TL;DR
GARFIELD learns a structured latent representation of possible scene kinematics from an image and optional sparse spatio-temporal constraints. The representation supports both joint trajectory sampling and direct point-wise density estimation.
Teaser
📝 Overview
Architecture
Goal Aware Representations of Future kInEmatic Latent Distributions (GARFIELD) represents future scene kinematics as localized motion distributions. Given an initial image and optional sparse constraints, a joint encoder produces spatio-temporal latents for scene elements over time. These latents encode uncertainty about where each element may move, and separate decoders expose that uncertainty as point samples, coherent trajectory rollouts, or probability heatmaps.
The full pipeline consists of four components:
Encoder: The joint encoder combines image features and sparse kinematic constraints into structured latents, each tied to one scene element and timestep.
Pointwise Decoder: The pointwise decoder trains the latents to represent localized distributions by sampling individual future track positions from each latent component.
Full Decoder: The full decoder samples complete trajectories jointly, preserving dependencies across points, timesteps, and scene elements for coherent motion realizations.
Density Decoder: The density decoder deterministically maps localized latents to probability heatmaps, enabling fast uncertainty inspection without Monte-Carlo sampling.
📊 Results
Motion Planning
EPE
GARFIELD infers accurate motion from very sparse information and, with only four conditioning points, outperforms Motion-I2V using sixteen.
Direct Density Decoding
Discrete Energy Scores
The density decoder achieves superior discrete energy scores while estimating motion densities orders of magnitude faster than Monte-Carlo sampling.
Entropy-informed Conditioning
Entropy-based cond
Selecting additional constraints by entropy collapses uncertainty without ground-truth errors and reaches comparable performance with fewer conditioning points.
🛠️ Usage
Weights and Inference Setup
For inference, you can clone the GitHub repository by running:
1@inproceedings{phan2026schrodingerscat,
2 title = {Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics},
3 author = {Phan, Timy and Wiese, Jannik and Ommer, Björn},
4 booktitle = {European Conference on Computer Vision (ECCV)},
5 year = {2026}
6}
