Views
No views yet
ControlNet is a neural network structure to control diffusion models by adding extra conditions. Official repository: https://github.com/lllyasviel/ControlNet
1{
2 "inputs": "A prompt used for image generation",
3 "negative_prompt": "low res, bad anatomy, worst quality, low quality",
4 "controlnet_type": "depth",
5 "image": "base64 encoded image"
6}controlnet_type are: canny_edge, pose, depth, scribble, segmentation, normal, hed, houghrequests.wget https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_imgvar/input_image_vermeer.png1import json
2from typing import List
3import requests as r
4import base64
5from PIL import Image
6from io import BytesIO
7
8ENDPOINT_URL = "" # your endpoint url
9HF_TOKEN = "" # your huggingface token `hf_xxx`
10
11# helper image utils
12def encode_image(image_path):
13 with open(image_path, "rb") as i:
14 b64 = base64.b64encode(i.read())
15 return b64.decode("utf-8")
16
17
18def predict(prompt, image, negative_prompt=None, controlnet_type = "normal"):
19 image = encode_image(image)
20
21 # prepare sample payload
22 request = {"inputs": prompt, "image": image, "negative_prompt": negative_prompt, "controlnet_type": controlnet_type}
23 # headers
24 headers = {
25 "Authorization": f"Bearer {HF_TOKEN}",
26 "Content-Type": "application/json",
27 "Accept": "image/png" # important to get an image back
28 }
29
30 response = r.post(ENDPOINT_URL, headers=headers, json=request)
31 if response.status_code != 200:
32 print(response.text)
33 raise Exception("Prediction failed")
34 img = Image.open(BytesIO(response.content))
35 return img
36
37
38prediction = predict(
39 prompt = "cloudy sky background lush landscape house and green trees, RAW photo (high detailed skin:1.2), 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3",
40 negative_prompt ="lowres, bad anatomy, worst quality, low quality, city, traffic",
41 controlnet_type = "hed",
42 image = "huggingface.png"
43)
44
45prediction.save("result.png")expected output
[Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543) by Lvmin Zhang and Maneesh Agrawala.
Using the pretrained models we can provide control images (for example, a depth map) to control Stable Diffusion text-to-image generation so that it follows the structure of the depth image and fills in the details.
The abstract of the paper is the following:
We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k). Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices. Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data. We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, segmentation maps, keypoints, etc. This may enrich the methods to control large diffusion models and further facilitate related applications.