These are controlnet weights trained on stabilityai/stable-diffusion-2-1-base with new type of conditioning.
You can find some example images below.
NOTE: This is a low precision model so image quality and charactersitics may be lagging
prompt: a white hoodie shirt on a size four model in a beach setting
DensePose Condition:
images_actual
Image Generated:
images_4)
prompt: a green jumper shirt and white pants with a green overcoat on top
images_actual
Image Generated:
images_5)
Intended uses & limitations
How to use
python
1from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
2from transformers import CLIPTokenizer, CLIPTextModel
3from PIL import Image
4import torch
56# Load pre-trained components7controlnet = ControlNetModel.from_pretrained("path/to/your/controlnet-model")8tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")9text_encoder = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14")1011pipe = StableDiffusionControlNetPipeline.from_pretrained(12"stabilityai/stable-diffusion-xl-base",13 controlnet=controlnet,14 tokenizer=tokenizer,15 text_encoder=text_encoder,16 torch_dtype=torch.float16,17).to("cuda")1819# Example: Generate an image of a red jacket on a model pose20pose_image = Image.open("path/to/pose.png").convert("RGB").resize((512,512))21prompt ="a red leather jacket with silver zippers, worn on a casual street-style model"2223image = pipe(prompt=prompt, control_image=pose_image, num_inference_steps=30).images[0]24image.save("output.png")25
Limitations and bias
Pose Alignment Errors: The model may fail to accurately align garments with extremely dynamic or occluded body poses.
Fabric Simulation: Lacks realistic physical behavior of fabrics like wrinkles, folds, or flowing movement.
Resolution Constraints: Default generation is 512×512. Upscaling may lose fidelity unless further post-processing is used.
Model Drift in Edge Cases: Struggles with rare combinations of garment types and unconventional descriptions.
Dataset Bias: DeepFashion and related datasets often overrepresent certain body types, genders, and skin tones, which can skew model generalization.
Style Bias: High fashion or Western clothing styles are more common in training data, leading to poorer performance for traditional or niche designs.
Recommendations:
Augment training data with underrepresented demographics and clothing styles
Use reinforcement or adversarial training to improve physics realism and fairness
Apply domain adaptation for traditional clothing categories
