Views
No views yet
MobileCLIP2-S4 matches the accuracy of SigLIP-SO400M/14 with 2x fewer parameters and surpasses DFN ViT-L/14 at 2.5x lower latency measured on iPhone12 Pro Max.MobileCLIP-S3/S4 are our new architectures trained on MobileCLIP’s training dataset, DataCompDR-1B (dashed lines).MobileCLIP-S0 obtains similar zero-shot performance as OpenAI's ViT-B/16 model while being 4.8x faster and 2.8x smaller.MobileCLIP-S2 obtains better avg zero-shot performance than SigLIP's ViT-B/16 model while being 2.3x faster and 2.1x smaller, and trained with 3x less seen samples.MobileCLIP-B (LT) attains zero-shot ImageNet performance of 77.2% which is significantly better than recent works like DFN and SigLIP with similar architectures or even OpenAI's ViT-L/14@336.| Model | # Seen Samples (B) | # Params (M) (img + txt) | Latency (ms) (img + txt) | IN-1k Zero-Shot Top-1 Acc. (%) | Avg. Perf. (%) on 38 datasets |
|---|---|---|---|---|---|
| MobileCLIP2-S0 | 13 | 11.4 + 63.4 | 1.5 + 3.3 | 71.5 | 59.7 |
| MobileCLIP2-S2 | 13 | 35.7 + 63.4 | 3.6 + 3.3 | 77.2 | 64.1 |
| MobileCLIP2-B | 13 | 86.3 + 63.4 | 10.4 + 3.3 | 79.4 | 65.8 |
| MobileCLIP2-S3 | 13 | 125.1 + 123.6 | 8.0 + 6.6 | 80.7 | 66.8 |
| MobileCLIP2-L/14 | 13 | 304.3 + 123.6 | 57.9 + 6.6 | 81.9 | 67.8 |
| MobileCLIP2-S4 | 13 | 321.6 + 123.6 | 19.6 + 6.6 | 81.9 | 67.5 |
| MobileCLIP-S0 | 13 | 11.4 + 42.4 | 1.5 + 1.6 | 67.8 | 58.1 |
| MobileCLIP-S1 | 13 | 21.5 + 63.4 | 2.5 + 3.3 | 72.6 | 61.3 |
| MobileCLIP-S2 | 13 | 35.7 + 63.4 | 3.6 + 3.3 | 74.4 | 63.7 |
| MobileCLIP-B | 13 | 86.3 + 63.4 | 10.4 + 3.3 | 76.8 | 65.2 |
| MobileCLIP-B (LT) | 36 | 86.3 + 63.4 | 10.4 + 3.3 | 77.2 | 65.8 |
| MobileCLIP-S3 | 13 | 125.1 + 123.6 | 8.0 + 6.6 | 78.3 | 66.3 |
| MobileCLIP-L/14 | 13 | 304.3 + 123.6 | 57.9 + 6.6 | 79.5 | 66.9 |
| MobileCLIP-S4 | 13 | 321.6 + 123.6 | 19.6 + 6.6 | 79.4 | 68.1 |
Files and versions tab, and download the PyTorch checkpoint.
For programmatic downloading, if you have huggingface_hub installed, you can also run:hf download apple/MobileCLIP2-S4ml-mobileclip by following the instructions in the repo. It uses an API similar to open_clip's.
You can run inference with a code snippet like the following:1import torch
2import open_clip
3from PIL import Image
4from mobileclip.modules.common.mobileone import reparameterize_model
5
6model, _, preprocess = open_clip.create_model_and_transforms('MobileCLIP2-S4', pretrained='/path/to/mobileclip2_s4.pt')
7tokenizer = open_clip.get_tokenizer('MobileCLIP2-S4')
8
9# Model needs to be in eval mode for inference because of batchnorm layers unlike ViTs
10model.eval()
11
12# For inference/model exporting purposes, please reparameterize first
13model = reparameterize_model(model)
14
15image = preprocess(Image.open("docs/fig_accuracy_latency.png").convert('RGB')).unsqueeze(0)
16text = tokenizer(["a diagram", "a dog", "a cat"])
17
18with torch.no_grad(), torch.cuda.amp.autocast():
19 image_features = model.encode_image(image)
20 text_features = model.encode_text(text)
21 image_features /= image_features.norm(dim=-1, keepdim=True)
22 text_features /= text_features.norm(dim=-1, keepdim=True)
23
24 text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
25
26print("Label probs:", text_probs)