import torch
from diffusers import StableDiffusionPipeline
from peft import PeftModel

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
    safety_checker=None,
)

# Merge the adapter at 0.6 strength (see "Adapter strength" below)
peft_unet = PeftModel.from_pretrained(pipe.unet, "MayankTamakuwala/sd15-wikiart-impressionism-lora")
for module in peft_unet.modules():
    if hasattr(module, "scaling") and isinstance(module.scaling, dict):
        for name in module.scaling:
            module.scaling[name] *= 0.6
pipe.unet = peft_unet.merge_and_unload()
pipe = pipe.to("cuda")

image = pipe(
    "an Impressionism painting, landscape, by Claude Monet",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]
image.save("output.png")

Adapter strength

Use ~0.6, not 1.0. At full strength this adapter imposes palette and brushwork hard enough to dissolve composition in complex multi-figure scenes. A matched-seed sweep over 0.4 / 0.6 / 0.8 / 1.0 put the breakdown between 0.6 and 0.8, while the learned behaviours survive down to 0.4.

Prompting

Trained on captions synthesized from WikiArt metadata in the form:

an Impressionism painting, {genre}, by {artist}

Prompts in that shape work best, e.g. an Impressionism painting, landscape, by Claude Monet. Free-form prompts also work; the style transfers to novel compositions.

What the adapter learned

Two effects are clearly attributable to fine-tuning, since neither is prompted and neither appears in base SD 1.5 output at matched seeds:

No picture frames. Base SD renders "a painting" as a photograph of a framed painting on a wall. WikiArt images are cropped to the canvas, so the adapter produces edge-to-edge artwork.
Photographed-painting colour. Output shifts toward the muted, slightly aged palette of real scanned paintings, away from SD's idealised saturation.

Evaluation

1,000 generated images against 653 held-out WikiArt Impressionism images the model never saw (seeded train/val split):

Model	FID ↓	CLIP ↑
Base SD 1.5	106.50	0.3141
+ this LoRA @ 1.0	102.55	0.3037
+ this LoRA @ 0.6	100.38	0.3092

FID is computed with clean-fid; CLIP score with OpenCLIP ViT-B-32. Note the absolute FID is inflated by the small reference set (653 images, below the ~1000 where FID stabilises) -- the delta against base is the meaningful quantity, not the absolute value.

Training


Base model	`runwayml/stable-diffusion-v1-5`
Dataset	`huggan/wikiart`, Impressionism, 5,000 images
LoRA rank / alpha	16 / 16
Target modules	`to_q`, `to_k`, `to_v`, `to_out.0`
Trainable params	3,188,736 (0.37% of 862,709,700)
Epochs / steps	6 / 3,750
Effective batch	8
Optimizer	AdamW, lr 1e-4, cosine schedule
Precision	bf16
Hardware	1× NVIDIA H200, 12m41s

Source

Training, evaluation, and serving code: https://github.com/MayankTamakuwala/Diffusion-Based-Image-Generation-Editing

Limitations

Complex multi-figure scenes degrade at high adapter strength.
Trained on one style; other WikiArt styles need their own adapter.
Inherits Stable Diffusion 1.5's limitations and biases.
Non-commercial research use, per the CreativeML OpenRAIL-M licence.