-
Training epochs: 4
-
Training steps: 100
-
Learning rate: 0.0001
- Learning rate schedule: constant
- Warmup steps: 0
-
Max grad value: 2.0
-
Effective batch size: 1
- Micro-batch size: 1
- Gradient accumulation steps: 1
- Number of GPUs: 1
-
Gradient checkpointing: True
-
Prediction type: epsilon (extra parameters=['training_scheduler_timestep_spacing=trailing', 'inference_scheduler_timestep_spacing=trailing'])
-
Optimizer: bnb-lion8bit
-
Trainable parameter precision: Pure BF16
-
Base model precision: no_change
-
Caption dropout probability: 0.1%
-
LoRA Rank: 128
-
LoRA Alpha: 128.0
-
LoRA Dropout: 0.1
-
LoRA initialisation style: default
1import torch
2from diffusers import DiffusionPipeline
3
4model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
5adapter_id = 'bghira/simpletuner-controlnet-sdxl-lora-test'
6pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16
7pipeline.load_lora_weights(adapter_id)
8
9prompt = "A photo-realistic image of a cat"
10negative_prompt = 'blurry, cropped, ugly'
11
12## Optional: quantise the model to save on vram.
13## Note: The model was not quantised during training, so it is not necessary to quantise it during inference time.
14#from optimum.quanto import quantize, freeze, qint8
15#quantize(pipeline.unet, weights=qint8)
16#freeze(pipeline.unet)
17
18pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level
19model_output = pipeline(
20 prompt=prompt,
21 negative_prompt=negative_prompt,
22 num_inference_steps=20,
23 generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42),
24 width=1024,
25 height=1024,
26 guidance_scale=4.2,
27 guidance_rescale=0.0,
28).images[0]
29
30model_output.save("output.png", format="PNG")
31
