Hongfei Zhang ^1*, Kanghao Chen ^1,5*, Zixin Zhang ^1,5, Harold H. Chen ^1,5, Yuanhuiyi Lyu ¹, Yuqi Zhang ³, Shuai Yang ¹, Kun Zhou ⁴, Ying-Cong Chen ^1,2,✉

¹ HKUST(GZ) ² HKUST ³ Fudan University ⁴ Shenzhen University ⁵ Knowin

* Equal Contribution. ✉Corresponding author.

✅ 2025.11 — Released inference pipeline & demo dataset ✔️

✅ 2025.11 — Uploaded official DualCamCtrl checkpoints to HuggingFace 🔑

⬜ Release the training code 🚀

This paper presents DualCamCtrl, a novel end-to-end diffusion model for camera-controlled video generation. Recent works have advanced this field by representing camera poses as ray-based conditions, yet they often lack sufficient scene understanding and geometric awareness. DualCamCtrl specifically targets this limitation by introducing a dual-branch framework that mutually generates camera-consistent RGB and depth sequences. To harmonize these two modalities, we further propose the SemantIc Guided Mutual Alignment (SIGMA) mechanism, which performs RGB–depth fusion in a semantics-guided and mutually reinforced manner. These designs collectively enable DualCamCtrl to better disentangle appearance and geometry modeling, generating videos that more faithfully adhere to the specified camera trajectories. Extensive experiments demonstrate that DualCamCtrl achieves more consistent camera-controlled video generation with over 40% reduction on camera motion errors compared with prior methods.

Comparison between our method and other state-of-the-art approaches. Given the same camera pose and input image as generation conditions, our method achieves the best alignment between camera motion and scene dynamics, producing the most visually accurate video. The ’+’ signs marked in the figure serve as anchors for visual comparison.

Quantitative comparisons on I2V setting. ↑ / ↓ denotes higher/lower is better. Best and second best results highlighted.

Quantitative comparisons on T2V setting across REALESTATE10K and DL3DV.

git clone https://github.com/soyouthinkyoucantell/DualCamCtrl.git
conda create -n dualcamctrl python=3.11
conda activate dualcamctrl

cd DualCamCtrl
pip install -e .

mkdir dependency
cd dependency
git clone https://github.com/rmbrualla/pycolmap.git 
cd pycolmap
pip install -e .
pip install numpy==1.26.4 peft accelerate==1.9.0 decord==0.6.0 deepspeed diffusers omegaconf  

Get the checkpoints from the HuggingFace repo:  DualCamCtrl Checkpoints

Your project structure should be like

DualCamCtrl/
├── checkpoints/                 # ← Put downloaded .pt here
│   └── dualcamctrl_diffusion_transformer.pt
├── demo_dataset/                # Small demo dataset strcture
├── demo_pic/                    # Demo images for quick inference
├── diffsynth/                   
├── examples/   
├── ....                 
├── requirements.txt             
├── README.md                    
└── setup.py                    

cd .. # make sure you are at the root dir 
export PYTHONPATH=.
python -m test_script.test_demo