Video Mask2Former model trained on YouTubeVIS-2021 instance segmentation (tiny-sized version, Swin backbone). It was introduced in the paper
Mask2Former for Video Instance Segmentation
and first released in
this repository.
Video Mask2Former is an extension of the original Mask2Former paper released under the name,
Masked-attention Mask Transformer for Universal Image Segmentation.
Disclaimer: The team releasing Mask2Former did not write a model card for this model so this model card has been written by the Hugging Face team.
Mask2Former addresses instance, semantic and panoptic segmentation with the same paradigm: by predicting a set of masks and corresponding labels. Hence, all 3 tasks are treated as if they were instance segmentation. Mask2Former outperforms the previous SOTA,
MaskFormer both in terms of performance an efficiency by (i) replacing the pixel decoder with a more advanced multi-scale deformable attention Transformer, (ii) adopting a Transformer decoder with masked attention to boost performance without
without introducing additional computation and (iii) improving training efficiency by calculating the loss on subsampled points instead of whole masks.
In the paper
Mask2Former for Video Instance Segmentation
, the authors have shown that Mask2Former also achieves state-of-the-art performance on video instance segmentation without modifying the architecture, the loss or even the training pipeline.
You can use this particular checkpoint for instance segmentation. See the
model hub to look for other fine-tuned versions of this model that may interest you.
1import torch
2import torchvision
3from huggingface_hub import hf_hub_download
4from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation
5
6
7# load Mask2Former trained on YouTubeVIS 2021 instance segmentation
8processor = AutoImageProcessor.from_pretrained("facebook/video-mask2former-swin-tiny-youtubevis-2021-instance")
9model = Mask2FormerForUniversalSegmentation.from_pretrained("facebook/video-mask2former-swin-tiny-youtubevis-2021-instance")
10
11file_path = hf_hub_download(repo_id="shivi/video-demo", filename="cars.mp4", repo_type="dataset")
12video = torchvision.io.read_video(file_path)[0]
13video_frames = [image_processor(images=frame, return_tensors="pt").pixel_values for frame in video]
14video_input = torch.cat(video_frames)
15
16with torch.no_grad():
17 outputs = model(**video_input)
18
19# model predicts class_queries_logits of shape `(batch_size, num_queries, num_classes)`
20# and masks_queries_logits of shape `(num_queries, batch_size, height, width)`
21class_queries_logits = outputs.class_queries_logits
22masks_queries_logits = outputs.masks_queries_logits
23
24# you can pass them to processor for postprocessing
25result = image_processor.post_process_video_instance_segmentation(outputs, target_sizes=[tuple(video.shape[1:3])])[0]
26# we refer to the demo notebooks for visualization (see "Resources" section in the Mask2Former docs)
27predicted_video_instance_map = result["segmentation"]
For more code examples, we refer to the
documentation.
