DreamOmni2: Multimodal Instruction-based Editing and Generation
This repository contains DreamOmni2, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in multimodal instruction-based editing and generation. These tasks support both text and image instructions and extend the scope to include both concrete and abstract concepts, greatly enhancing their practical applications.
For traditional subject-driven generation based on concrete objects, DreamOmni2 achieves the best results among open-source models, showing superior identity and pose consistency. Additionally, DreamOmni2 can reference abstract attributes (such as material, texture, makeup, hairstyle, posture, design style, artistic style, etc.), even surpassing commercial models in this area.
(2) Multimodal Instruction-based Editing
Beyond traditional instruction-based editing models, DreamOmni2 supports multimodal instruction editing. In everyday editing tasks, there are often elements that are difficult to describe purely with language and require reference images. Our model addresses this need, supporting references to any concrete objects and abstract attributes, with performance comparable to commercial models.
(3) Unified Generation and Editing Model
Building upon these two new tasks, we introduce DreamOmni2, which is capable of multimodal instruction-based editing and generation under any concrete or abstract concept guidance. Overall, DreamOmni2 is a more intelligent and powerful open-sourced unified generation and editing model, offering enhanced capabilities across a wide range of tasks.
Editing and Generation Model?
Editing and generation are distinct tasks. Editing requires strict consistency in preserving the non-edited areas of the source image, while generation only needs to retain the ID, IP, or attribution from the reference image as per the instructions, allowing the entire image to be regenerated with a focus on aesthetics. We’ve found that the instructions for generation and editing are often similar, so we’ve separated these two tasks to make it easier for users to choose the appropriate task type.
Notably, for editing tasks, due to the format settings of the training data, we need to place the image to be edited in the first position.
python
1python3 /mnt/bn/unifygen/xiabin_dev/iclr2026/DreamOmni2/inference_edit.py \
2--input_img_path "example_input/edit_tests/src.jpg""example_input/edit_tests/ref.jpg" \
3--input_instruction "Make the woman from the second image stand on the road in the first image." \
4--output_path "example_input/edit_tests/edit_res.png"
Multimodal Instruction-based Generation
python
1python3 /mnt/bn/unifygen/xiabin_dev/iclr2026/DreamOmni2/inference_gen.py \
2--input_img_path "example_input/gen_tests/img1.jpg""example_input/gen_tests/img2.jpg" \
3--input_instruction "In the scene, the character from the first image stands on the left, and the character from the second image stands on the right. They are shaking hands against the backdrop of a spaceship interior." \
4--output_path "example_input/gen_tests/gen_res.png" \
5--height 1024 \
6--width 1024
This project strives to impact the domain of AI-driven image generation positively. Users are granted the freedom to
create images using this tool, but they are expected to comply with local laws and utilize it responsibly.
The developers do not assume any responsibility for potential misuse by users.
Citation
If DreamOmni2 is helpful, please help to ⭐ the repo.
If you find this project useful for your research, please consider citing our paper.
bibtex
1@article{xia2025dreamomni2,
2 title={DreamOmni2: Multimodal Instruction-based Editing and Generation},
3 author={Bin Xia and Binbin Liu and Haokun Lin and Zhichao Wei and Haifeng Shi and Rui Wang and Sijie Yan and Hongsheng Li and Guanglu Song and Yanwei Fu},
4 year={2025},
5 eprint={2510.06679},
6 archivePrefix={arXiv},
7 primaryClass={cs.CV},
8 url={https://arxiv.org/abs/2510.06679},
9}
