🎬 CoInteract

 

 

 

✨ Highlights

CoInteract is the first end-to-end framework that generates physically-consistent human-object interaction (HOI) videos with zero additional inference cost. Given a person image, a product image, text prompts, and optional speech audio, CoInteract produces realistic videos where humans naturally grasp, wear, present, and manipulate objects — with no hand-object interpenetration or geometric misalignment.

• 🏆 State-of-the-art on HOI video synthesis benchmarks
• 🤝 Physically plausible hand-object contact (significantly reduced interpenetration)
• ⚡ Zero inference overhead — auxiliary HOI branch removed at test time
• 🎯 Supports diverse interactions: grasping, wearing, presenting, carrying, and more
• 🛒 Real-world ready: Virtual Try-On, Digital Human Commerce, Physics Simulation

🏗️ Architecture

• 🧠 Human-Aware MoE — Spatially-supervised Mixture-of-Experts routes tokens to region-specialized experts (Head, Hand, Base), ensuring high structural fidelity for hands and faces with minimal parameter overhead.
• 🔗 Dual-Stream Co-Generation — An auxiliary HOI structure stream is jointly trained with the RGB stream within a shared DiT backbone, forcing the model to learn spatial and interaction relationships.
• ✨ Asymmetric Co-Attention — A two-stage training strategy with asymmetric attention masks embeds physical interaction rules, enabling the HOI branch to be completely removed at inference with zero overhead.

🎬 Demo

• 🛍️ Digital Human Commerce — AI-powered digital humans presenting products in live-stream e-commerce
• 👗 Virtual Try-On — Physically realistic garment and accessory interactions
• ⚙️ Physics Simulation — Generating high-quality HOI training data for robotics

🔧 Model Details

📊 Quantitative Results

Full quantitative results will be released upon paper acceptance.

📝 Citation

🙏 Acknowledgements