These are LoRA adaption weights for stabilityai/stable-diffusion-xl-base-1.0 trained on @fffiloni's SD-XL trainer.
The weights were trained on the concept prompt:
Use this keyword to trigger your custom model in your prompts.
LoRA for the text encoder was enabled: False.
Special VAE used for training: madebyollin/sdxl-vae-fp16-fix.
In addition make sure to install transformers, safetensors, accelerate as well as the invisible watermark:
1import torch
2from diffusers import DiffusionPipeline, AutoencoderKL
3device = "cuda" if torch.cuda.is_available() else "cpu"
4vae = AutoencoderKL.from_pretrained('madebyollin/sdxl-vae-fp16-fix', torch_dtype=torch.float16)
5pipe = DiffusionPipeline.from_pretrained(
6 "stabilityai/stable-diffusion-xl-base-1.0",
7 vae=vae, torch_dtype=torch.float16, variant="fp16",
8 use_safetensors=True
9)
10pipe.to(device)
11# This is where you load your trained weights
12specific_safetensors = "pytorch_lora_weights.safetensors"
13lora_scale = 0.9
14pipe.load_lora_weights(
15 'amehfooz/lora',
16 weight_name = specific_safetensors,
17 # use_auth_token = True
18)
19prompt = "A majestic ruwaifarufy jumping from a big stone at night"
20image = pipe(
21 prompt=prompt,
22 num_inference_steps=50,
23 cross_attention_kwargs={"scale": lora_scale}
24).images[0]
