These are ControlNet-XS weights trained on
stabilityai/stable-diffusion-xl-base-1.0 and
stabilityai/stable-diffusion-2-1 on edge and depthmap conditioning respectively. You can find more details and further visual examples on the project page
ControlNet-XS.
The code is based on on the StableDiffusion frameworks. To use the ControlNet-XS, you need to access the weights for the StableDiffusion version that you want to control separately.
We provide the weights with both depth and edge control for StableDiffusion2.1 and StableDiffusion-XL.
After obtaining the weights, you need the replace the paths to the weights of StableDiffusion and ControlNet-XS in the config files.
1import scripts.control_utils as cu
2import torch
3from PIL import Image
4
5path_to_config = 'ControlNet-XS-main/configs/inference/sdxl/sdxl_encD_canny_48m.yaml'
6model = cu.create_model(path_to_config).to('cuda')
7
8image_path = 'PATH/TO/IMAGES/Shoe.png'
9
10canny_high_th = 250
11canny_low_th = 100
12size = 768
13num_samples=2
14
15image = cu.get_image(image_path, size=size)
16edges = cu.get_canny_edges(image, low_th=canny_low_th, high_th=canny_high_th)
17
18samples, controls = cu.get_sdxl_sample(
19 guidance=edges,
20 ddim_steps=10,
21 num_samples=num_samples,
22 model=model,
23 shape=[4, size // 8, size // 8],
24 control_scale=0.95,
25 prompt='cinematic, shoe in the streets, made from meat, photorealistic shoe, highly detailed',
26 n_prompt='lowres, bad anatomy, worst quality, low quality',
27)
28
29
30Image.fromarray(cu.create_image_grid(samples)).save('SDXL_MyShoe.png')
1import scripts.control_utils as cu
2import torch
3from PIL import Image
4
5path_to_config = 'PATH/TO/CONFIG/sd21_encD_depth_14m.yaml'
6model = cu.create_model(path_to_config).to('cuda')
7
8size = 768
9image_path = 'PATH/TO/IMAGES/Shoe.png'
10
11
12image = cu.get_image(image_path, size=size)
13depth = cu.get_midas_depth(image, max_resolution=size)
14num_samples = 2
15
16samples, controls = cu.get_sd_sample(
17 guidance=depth,
18 ddim_steps=10,
19 num_samples=num_samples,
20 model=model,
21 shape=[4, size // 8, size // 8],
22 control_scale=0.95,
23 prompt='cinematic, advertising shot, shoe in a city street, photorealistic shoe, colourful, highly detailed',
24 n_prompt='low quality, bad quality, sketches'
25)
26
27
28Image.fromarray(cu.create_image_grid(samples)).save('SD_MyShoe.png')
