Views
No views yet
DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation
Xu Guo * , Fulong Ye * , Qichao Sun *†, Liyang Chen, Bingchuan Li †, Pengze Zhang, Jiawei Liu, Songtao Zhao §, Qian He, Xiangwang Hou §
* Equal contribution, † Project lead, § Corresponding author
Tsinghua University | Intelligent Creation Team, ByteDance
1python3 download_weights.py
2conda create -n dreamid_omni python=3.11
3conda activate dreamid_omni
4pip install torch==2.6.0 torchvision torchaudio
5pip install -r requirements.txt
6pip install flash_attn --no-build-isolationpython3 inference_r2av.py --config-file dreamid_omni/configs/inference/inference_r2av.yamltorchrun --nnodes 1 --nproc_per_node 8 inference_r2av.py --config-file dreamid_omni/configs/inference/inference_r2av.yamldreamid_omni/configs/inference/inference_r2av.yaml and set sp_size: 8<sub1>, <sub2> - Represents the character IPs provided in your input images (e.g., <img1> corresponds to <sub1>). Use these tags in your prompt to specify who is acting or speaking.<S>Your speech content here<E> - Text enclosed in these tags will be converted to speech using the corresponding character's reference audio.test_case/oneiptest_case/twoip@misc{guo2026dreamidomni,
title={DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation},
author={Xu Guo and Fulong Ye and Qichao Sun and Liyang Chen and Bingchuan Li and Pengze Zhang and Jiawei Liu and Songtao Zhao and Qian He and Xiangwang Hou},
year={2026},
eprint={2602.12160},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2602.12160},
}