OminiControl

OminiControl: Minimal and Universal Control for Diffusion Transformer
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang
xML Lab, National University of Singapore

OminiControl2: Efficient Conditioning for Diffusion Transformers
Zhenxiong Tan, Qiaochu Xue, Xingyi Yang, Songhua Liu, and Xinchao Wang
xML Lab, National University of Singapore

Features

• Universal Control 🌐: A unified control framework that supports both subject-driven control and spatial control (such as edge-guided and in-painting generation).
• Minimal Design 🚀: Injects control signals while preserving original model structure. Only introduces 0.1% additional parameters to the base model.

News

• 2025-05-12: ⭐️ The code of OminiControl2 is released. It introduces a new efficient conditioning method for diffusion transformers. (Check out the training code here).
• 2025-05-12: Support custom style LoRA. (Check out the example).
• 2025-04-09: ⭐️ OminiControl Art is released. It can stylize any image with a artistic style. (Check out the demo and inference examples).
• 2024-12-26: Training code are released. Now you can create your own OminiControl model by customizing any control tasks (3D, multi-view, pose-guided, try-on, etc.) with the FLUX model. Check the training folder for more details.

Quick Start

Setup (Optional)

1. Environment setup

1. Requirements installation

Usage example

1. Subject-driven generation: examples/subject.ipynb
2. In-painting: examples/inpainting.ipynb
3. Canny edge to image, depth to image, colorization, deblurring: examples/spatial.ipynb

Guidelines for subject-driven generation

1. Input images are automatically center-cropped and resized to 512x512 resolution.
2. When writing prompts, refer to the subject using phrases like this item, the object, or it. e.g.
   1. A close up view of this item. It is placed on a wooden table.
   2. A young lady is wearing this shirt.
3. The model primarily works with objects rather than human subjects currently, due to the absence of human data in training.

Generated samples

Subject-driven generation

Spatially aligned control

1. Image Inpainting (Left: original image; Center: masked image; Right: filled image)

• Prompt: The Mona Lisa is wearing a white VR headset with 'Omini' written on it.
  

  

• Prompt: A yellow book with the word 'OMINI' in large font on the cover. The text 'for FLUX' appears at the bottom.
  

  

1. Other spatially aligned tasks (Canny edge to image, depth to image, colorization, deblurring)
   
   Click to show

   

   

   

   

   Prompt: A light gray sofa stands against a white wall, featuring a black and white geometric patterned pillow. A white side table sits next to the sofa, topped with a white adjustable desk lamp and some books. Dark hardwood flooring contrasts with the pale walls and furniture.

Stylize images

Models

Community Extensions

• ComfyUI-Diffusers-OminiControl - ComfyUI integration by @Macoron
• ComfyUI_RH_OminiControl - ComfyUI integration by @HM-RunningHub

Limitations

1. The model's subject-driven generation primarily works with objects rather than human subjects due to the absence of human data in training.
2. The subject-driven generation model may not work well with FLUX.1-dev.
3. The released model only supports the resolution of 512x512.

Training

To-do

• Release the training code.
• Release the model for higher resolution (1024x1024).

Acknowledgment

Citation

@article{tan2025ominicontrol,
  title={OminiControl: Minimal and Universal Control for Diffusion Transformer},
  author={Tan, Zhenxiong and Liu, Songhua and Yang, Xingyi and Xue, Qiaochu and Wang, Xinchao},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2025}
}

@article{tan2025ominicontrol2,
  title={OminiControl2: Efficient Conditioning for Diffusion Transformers},
  author={Tan, Zhenxiong and Xue, Qiaochu and Yang, Xingyi and Liu, Songhua and Wang, Xinchao},
  journal={arXiv preprint arXiv:2503.08280},
  year={2025}
}