A (kinda failed) attempt to train an SDXL LoRA model upon on a mixed style dataset of images from Dreamworks, Disney and Pixar. I have uploaded this Blip2-Captioned Dataset to Huggingface.

I usually prefer to train in LoRA ranks 4-16 because so far it always seemed to be enough and provided a reasonably small checkpoint but in this case, I only started getting results when I went for Rank-32, even tried 24 and it was not enough.

My conclusion is that it should have probably been trained as several different lower ranked LoRAs each for the specific visual sub-style and it would have probably looked way better for each.

Trigger words

You should use DreamDisPix style to trigger the image generation.

Download model

Weights for this model are available in Safetensors format.

Download them in the Files & versions tab.

Use it with the 🧨 diffusers library

from diffusers import AutoPipelineForText2Image
import torch

pipeline = AutoPipelineForText2Image.from_pretrained('stabilityai/stable-diffusion-xl-base-1.0', torch_dtype=torch.float16).to('cuda')
pipeline.load_lora_weights('Norod78/dream-dis-pix-xl', weight_name='SDXL-DreamDisPix-Lora-r32.safetensors')
image = pipeline('American gothic  DreamDisPix style  ').images[0]

For more details, including weighting, merging and fusing LoRAs, check the documentation on loading LoRAs in diffusers