This checkpoint is uploaded by Jiarui Xu.
The GroupViT model was proposed in
GroupViT: Semantic Segmentation Emerges from Text Supervision by Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang.
Inspired by
CLIP, GroupViT is a vision-language model that can perform zero-shot semantic segmentation on any given vocabulary categories.
Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.
1from PIL import Image
2import requests
3from transformers import AutoProcessor, GroupViTModel
4
5model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
6processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
7
8url = "http://images.cocodataset.org/val2017/000000039769.jpg"
9image = Image.open(requests.get(url, stream=True).raw)
10
11inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)
12
13outputs = model(**inputs)
14logits_per_image = outputs.logits_per_image # this is the image-text similarity score
15probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
The model was trained on publicly available image-caption data. This was done through a combination of crawling a handful of websites and using commonly-used pre-existing image datasets such as
YFCC100M. A large portion of the data comes from our crawling of the internet. This means that the data is more representative of people and societies most connected to the internet which tend to skew towards more developed nations, and younger, male users.
For more code examples, we refer to the
documentation.
1@article{xu2022groupvit,
2 author = {Xu, Jiarui and De Mello, Shalini and Liu, Sifei and Byeon, Wonmin and Breuel, Thomas and Kautz, Jan and Wang, Xiaolong},
3 title = {GroupViT: Semantic Segmentation Emerges from Text Supervision},
4 journal = {arXiv preprint arXiv:2202.11094},
5 year = {2022},
6}
