Pretrained CLIP-Vision-BERT pre-trained on translated Conceptual-12M image-text pairs using a masked language modeling (MLM) objective. 10M cleaned image-text pairs are translated using mBART-50 one-to-many model to 2.5M examples each in English, French, German and Spanish. This model is based on the VisualBERT which was introduced in
this paper and first released in
this repository. We trained CLIP-Vision-BERT model during community week hosted by Huggingface 🤗 using JAX/Flax.
This checkpoint is pre-trained for 60k steps.
Model description
CLIP-Vision-BERT is a modified BERT model which takes in visual embeddings from CLIP-Vision transformer and concatenates them with BERT textual embeddings before passing them to the self-attention layers of BERT. This is done for deep cross-modal interaction between the two modes.
Intended uses & limitations❗️
You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
Note that this model is primarily aimed at being fine-tuned on tasks such as visuo-linguistic sequence classification or visual question answering. We used this model to fine-tuned on a multi-translated version of the visual question answering task - VQA v2. Since Conceptual-12M is a dataset scraped from the internet, it will involve some biases which will also affect all fine-tuned versions of this model.
How to use❓
You can use this model directly with a pipeline for masked language modeling. You will need to clone the model from here. An example of usage is shown below:
python
1>>>from torchvision.io import read_image
2>>>import numpy as np
3>>>import os
4>>>from transformers import CLIPProcessor, BertTokenizerFast
5>>>from model.flax_clip_vision_bert.modeling_clip_vision_bert import FlaxCLIPVisionBertForMaskedLM
6>>> image_path = os.path.join('images/val2014', os.listdir('images/val2014')[0])7>>> img = read_image(image_path)8>>> clip_processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')9ftfy or spacy isnot installed using BERT BasicTokenizer instead of ftfy.10>>> clip_outputs = clip_processor(images=img)11>>> clip_outputs['pixel_values'][0]= clip_outputs['pixel_values'][0].transpose(1,2,0)# Need to transpose images as model expected channel last images.12>>> tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-uncased')13>>> model = FlaxCLIPVisionBertForMaskedLM.from_pretrained('flax-community/clip-vision-bert-cc12m-60k')14>>> text ="Three teddy [MASK] in a showcase."15>>> tokens = tokenizer([text], return_tensors="np")16>>> pixel_values = np.concatenate([clip_outputs['pixel_values']])17>>> outputs = model(pixel_values=pixel_values,**tokens)18>>> indices = np.where(tokens['input_ids']==tokenizer.mask_token_id)19>>> preds = outputs.logits[indices][0]20>>> sorted_indices = np.argsort(preds)[::-1]# Get reverse sorted scores21/home/crocoder/anaconda3/lib/python3.8/site-packages/jax/_src/numpy/lax_numpy.py:4615: UserWarning:'kind' argument to argsort is ignored.22 warnings.warn("'kind' argument to argsort is ignored.")23>>> top_5_indices = sorted_indices[:5]24>>> top_5_tokens = tokenizer.convert_ids_to_tokens(top_5_indices)25>>> top_5_scores = preds[top_5_indices]26>>>print(dict(zip(top_5_tokens, top_5_scores)))27{'bears':19.241959,'bear':17.700356,'animals':14.368396,'girls':14.343797,'dolls':14.274415}28
Training data 🏋🏻♂️
The CLIP-Vision-BERT model was pre-trained on a translated version of the Conceptual-12m dataset in four languages using mBART-50: English, French, German and Spanish, with 2.5M image-text pairs in each.
Though the original dataset contains 12M image-text pairs, a lot of the URLs are invalid now, and in some cases, images are corrupt or broken. We remove such examples from our data, which leaves us with approximately 10M image-text pairs.
Splits
We used 99% of the 10M examples as a train set, and the remaining ~ 100K examples as our validation set.
Training procedure 👨🏻💻
Preprocessing
The texts are lowercased and tokenized using WordPiece and a shared vocabulary size of approximately 110,000. The beginning of a new document is marked with [CLS] and the end of one by [SEP]
The details of the masking procedure for each sentence are the following:
15% of the tokens are masked.
In 80% of the cases, the masked tokens are replaced by [MASK].
In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
In the 10% remaining cases, the masked tokens are left as is.
The visual embeddings are taken from the CLIP-Vision model and combined with the textual embeddings inside the BERT embedding layer. The padding is done in the middle. Here is an example of what the embeddings look like:
A total length of 128 tokens, including the visual embeddings, is used. The texts are truncated or padded accordingly.
Pretraining
The checkpoint of the model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) 8 v3 TPU cores for 60k steps with a per device batch size of 64 and a max sequence length of 128. The optimizer used is Adafactor with a learning rate of 1e-4, learning rate warmup for 5,000 steps, and linear decay of the learning rate after.
The checkpoints for the fine-tuned model on this pre-trained checkpoint can be found here.
The fine-tuned model achieves eval accuracy of 49% on our validation dataset.
We thank Nilakshan Kunananthaseelan for helping us whenever he could get a chance. We also thank Abheesht Sharma for helping in the discussions in the initial phases. Luke Melas helped us get the CC-12M data on our TPU-VMs and we are very grateful to him.
This project would not be possible without the help of Patrick and Suraj who met with us frequently and helped review our approach and guided us throughout the project.
Huge thanks to Huggingface 🤗 & Google Jax/Flax team for such a wonderful community week and for answering our queries on the Slack channel, and for providing us with the TPU-VMs.
