Views
No views yet
ViT-B/32 image encoder. It should be used with clip_head.h5 from [my accompanying repo on GitHub]. Go to that repo for fully working example, and a simple usage example is as follows:1from transformers import AutoTokenizer, TFAutoModel
2import tensorflow as tf
3import numpy as np
4from PIL import Image
5import torch
6import clip
7
8model_name = "mys/distilbert-base-turkish-cased-clip"
9base_model = TFAutoModel.from_pretrained(model_name)
10tokenizer = AutoTokenizer.from_pretrained(model_name)
11head_model = tf.keras.models.load_model("./clip_head.h5")
12
13def encode_text(base_model, tokenizer, head_model, texts):
14 tokens = tokenizer(texts, padding=True, return_tensors='tf')
15 embs = base_model(**tokens)[0]
16
17 attention_masks = tf.cast(tokens['attention_mask'], tf.float32)
18 sample_length = tf.reduce_sum(attention_masks, axis=-1, keepdims=True)
19 masked_embs = embs * tf.expand_dims(attention_masks, axis=-1)
20 base_embs = tf.reduce_sum(masked_embs, axis=1) / tf.cast(sample_length, tf.float32)
21 clip_embs = head_model(base_embs)
22 clip_embs /= tf.norm(clip_embs, axis=-1, keepdims=True)
23 return clip_embs
24
25
26
27demo_images = {
28 "bilgisayarda çalışan bir insan": "myspc.jpeg",
29 "sahilde bir insan ve bir heykel": "mysdk.jpeg"
30 }
31
32clip_model, preprocess = clip.load("ViT-B/32")
33images = {key: Image.open(f"images/{value}") for key, value in demo_images.items()}
34img_inputs = torch.stack([preprocess(image).to('cpu') for image in images.values()])
35
36with torch.no_grad():
37 image_embs = clip_model.encode_image(img_inputs).float().to('cpu')
38
39image_embs /= image_embs.norm(dim=-1, keepdim=True)
40image_embs = image_embs.detach().numpy()
41text_embs = encode_text(base_model, tokenizer, head_model, list(images.keys())).numpy()
42similarities = image_embs @ text_embs.T
43logits = tf.nn.softmax(tf.convert_to_tensor(similarities)).numpy()
44idxs = np.argmax(logits, axis=-1).tolist()
45for i, (key, value) in enumerate(demo_images.items()):
46 print("path: ", value, "true label: ", key, "prediction: ", list(demo_images.keys())[idxs[i]], "score: ", logits[i, idxs[i]])images directory in the gitHub repo.encode_text() function agregates per-token hidden states outputted by the Distilbert model to produce a single vector per sequence. Then, clip_head.h5 model projects this vector onto the same vector space as CLIP's text encoder with a single dense layer. First, all the Distilbert layers were frozen an and the head dense layer was trained for a few epochs. Then, freezing was removed and the dense layer was trained with the Distilbert layers for a few more epochs. I created the dataset by machine-translating COCO captions into Turkish. During training, vector representations of English captions outputted by the original CLIP text encoder was used as target values, and MSE between these vectors and clip_head.h5 outputs were minimized.