CHATS is a next-generation framework that unifies human preference alignment with classifier-free guidance by modeling both preferred and dispreferred distributions and using a proxy-prompt-based sampling strategy for superior text–image alignment, fidelity, and aesthetic consistency. See the images generated below for examples.
-
Human-Aligned Fine-Tuning with CFG Integration
We integrate human preference alignment with classifier-free guidance sampling into a unified framework.
-
Proxy-Prompt Sampling
Leverage useful signals from both preferred and dispreferred distributions at test time.
-
Data Efficiency
State-of-the-art results across benchmarks with minimal fine-tuning effort on a small, high-quality dataset.
-
Plug-and-Play
Compatible with any diffusion backbone and existing guidance methods.
1git clone https://github.com/AIDC-AI/CHATS.git
2cd CHATS
3pip install -r requirements.txt
4
1import torch
2from pipelines.pipeline_sdxl import ChatsSDXLPipeline
3from pipelines.pipeline_sd15 import ChatsSD15Pipeline
4
5# Load CHATS-SDXL pipeline
6pipe = ChatsSDXLPipeline.from_pretrained(
7 "AIDC-AI/CHATS",
8 torch_dtype=torch.bfloat16
9)
10
11# or
12# Load CHATS-SD1.5 pipeline
13# pipe = ChatsSD15Pipeline.from_pretrained(
14# "AIDC-AI/CHATS-SD1d5",
15# torch_dtype=torch.bfloat16
16# )
17
18pipe.to("cuda")
19
20# Generate images
21images = pipe(
22 prompt=["A serene mountain lake at sunset"],
23 num_inference_steps=50,
24 guidance_scale=5,
25 seed=0
26)
27
28# Save outputs
29for i, img in enumerate(images):
30 img.save(f"output_{i}.png")
1
2
3accelerate launch --config_file=config/ac_ds_8gpu_zero0.yaml train.py \
4 --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
5 --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
6 --resolution=1024 \
7 --dataloader_num_workers 16 \
8 --train_batch_size=1 \
9 --gradient_accumulation_steps=16 \
10 --max_train_steps=6000 \
11 --learning_rate=3e-09 --scale_lr --lr_scheduler=constant_with_warmup --lr_warmup_steps=100 \
12 --mixed_precision=bf16 \
13 --allow_tf32 \
14 --checkpointing_steps=100 \
15 --output_dir=output \
16 --resume_from_checkpoint latest \
17 --use_adafactor \
18 --gradient_checkpointing \
19 --dataset_name=data-is-better-together/open-image-preferences-v1-binarized \
20 --sdxl \
1@inproceedings{fu2025chats,
2title={CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation},
3author={Fu, Minghao and Wang, Guo-Hua and Cao, Liangfu and Chen, Qing-Guo and Xu, Zhao and Luo, Weihua and Zhang, Kaifu},
4booktitle={International Conference on Machine Learning (ICML)},
5year={2025}
6}
The project is released under Apache License 2.0 (
http://www.apache.org/licenses/LICENSE-2.0, SPDX-License-identifier: Apache-2.0).
We used compliance checking algorithms during the training process, to ensure the compliance of the trained model to the best of our ability. Due to complex data and the diversity of language model usage scenarios, we cannot guarantee that the model is completely free of copyright issues or improper content. If you believe anything infringes on your rights or generates improper content, please contact us, and we will promptly address the matter.
