CVPR 2026 · Sriram Narayanan¹², Ziyu Jiang², Srinivasa G. Narasimhan¹, Manmohan Chandraker²³ (¹ Carnegie Mellon University · ² NEC Labs America · ³ UC San Diego)

Project Page · arXiv · Paper (PDF) · Simulation Code (PhyCo-Sim) · Dataset

Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes — without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.

phyco.pt is a single self-contained checkpoint built on the frozen NVIDIA Cosmos-Predict2-2B-Video2World DiT, with PhyCo's LoRA fine-tuning merged in and all three physics-conditioning ControlNet branches baked in:

Given a single input frame and pixel-aligned physical-property maps, PhyCo generates a short video whose motion follows the specified physics. No simulator or geometry reconstruction is used at inference.

This checkpoint is consumed by the PhyCo video-generation / evaluation code (see the project page for the code release). This repo contains:

Download both and run the canonical generation command:

--dynamic_controlnet selects the relevant branch per scenario (friction/restitution → controlnet_1, deformability → controlnet_2, force → controlnet_3).

This model is released under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. It is built on top of NVIDIA's Cosmos-Predict2, whose code/weights remain subject to their original Apache-2.0 / NVIDIA Open Model License terms.