Views
No views yet
CLIP is a contrastive model that learns to maximize the cosine similarity between a given image and caption, however, there is no guarantee that these embeddings are in the same space. While the embeddings generated are close the image and text embeddings occupy two disjoint sets.
1# Load Models
2clip_model = clip.load("ViT-L/14")
3decoder = Decoder(checkpoint="best.pth") # A decoder trained on CLIP Image embeddings
4
5# Retrieve prompt from user and encode with CLIP
6prompt = "A corgi wearing sunglasses"
7tokenized_text = tokenize(prompt)
8text_embedding = clip_model.encode_text(tokenized_text)
9
10# Now, pass the text embedding to the decoder
11predicted_image = decoder.sample(text_embedding)Question: Can you spot the issue here?Answer: We’re trying to generate an image from a text embedding!
1# Load Models
2prior= Prior(checkpoint="prior.pth") # A decoder trained to go from: text-> clip text emb -> clip img emb
3decoder = Decoder(checkpoint="decoder.pth") # A decoder trained on CLIP Image embeddings
4
5# Retrieve prompt from user and encode with a prior
6prompt = "A corgi wearing sunglasses"
7tokenized_text = tokenize(prompt)
8text_embedding = prior.sample(tokenized_text) # <-- now we get an embedding in the same space as images!
9
10# Now, pass the predicted image embedding to the decoder
11predicted_image = decoder.sample(text_embedding)You may be asking yourself the following question:"Why don't you just train the decoder on clip text embeddings instead of image embeddings?"OpenAI covers this topic in their DALLE-2 paper. The TL;DR is "it doesn't work as well as decoders trained on image embeddings"...also...its just an example :smile:
1import torch
2from dalle2_pytorch import DiffusionPrior, DiffusionPriorNetwork, OpenAIClipAdapter
3from dalle2_pytorch.trainer import DiffusionPriorTrainer
4
5def load_diffusion_model(dprior_path):
6
7 prior_network = DiffusionPriorNetwork(
8 dim=768,
9 depth=24,
10 dim_head=64,
11 heads=32,
12 normformer=True,
13 attn_dropout=5e-2,
14 ff_dropout=5e-2,
15 num_time_embeds=1,
16 num_image_embeds=1,
17 num_text_embeds=1,
18 num_timesteps=1000,
19 ff_mult=4
20 )
21
22 diffusion_prior = DiffusionPrior(
23 net=prior_network,
24 clip=OpenAIClipAdapter("ViT-L/14"),
25 image_embed_dim=768,
26 timesteps=1000,
27 cond_drop_prob=0.1,
28 loss_type="l2",
29 condition_on_text_encodings=True,
30
31 )
32
33 trainer = DiffusionPriorTrainer(
34 diffusion_prior=diffusion_prior,
35 lr=1.1e-4,
36 wd=6.02e-2,
37 max_grad_norm=0.5,
38 amp=False,
39 group_wd_params=True,
40 use_ema=True,
41 device=device,
42 accelerator=None,
43 )
44
45 trainer.load(dprior_path)
46
47 return trainer1# tokenize the text
2tokenized_text = clip.tokenize("<your amazing prompt>")
3# predict an embedding
4predicted_embedding = prior.sample(tokenized_text, n_samples_per_batch=2, cond_scale=1.0).sample() is of the same shape as your training data along the non-batch dimension(s). For example, a prior trained on ViT-L/14 embeddings will predict an embedding of shape (1, 768).For CLIP priors, this is quite handy as it means that you can use prior.sample(tokenizer_text) as a drop in replacement for clip.encode_text().
n=2). Put simply, the idea here is that you avoid getting unlucky with a bad embedding generation by creating two; and selecting the one with the higher cosine similarity with the prompt.1.0). It is unclear whether OpenAI uses a higher value for the prior specifically, or only on the decoder. Local testing has shown poor results with anything higher than 1.0 but ymmv.