Model Type: Image classification / feature backbone
Model Stats:
- Params (M): 86.6
- GMACs: 16.9
- Activations (M): 16.5
- Image size: 224 x 224
Papers:
- When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations: https://arxiv.org/abs/2106.01548
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: https://arxiv.org/abs/2010.11929v2
Dataset: ImageNet-1k
Original: https://github.com/google-research/vision_transformer

Model Usage

Image Classification

python

from urllib.request import urlopen

import jax
from PIL import Image

import jaxnn

img = Image.open(urlopen(
    "https://huggingface.co/datasets/huggingface/cats-image/resolve/main/cats_image.jpeg"
))

model = jaxnn.create_model("vit_base_patch16_224.sam_in1k", pretrained=True)
model.eval()

data_config = jaxnn.data.resolve_model_data_config(model)
transforms = jaxnn.data.create_transform(**data_config, is_training=False)

x = jax.numpy.expand_dims(transforms(img), 0)
output = model(x, deterministic=True)

top5_probabilities, top5_class_indices = jax.lax.top_k(
    jax.nn.softmax(output, axis=-1) * 100,
    k=5,
)

Image Embeddings

python

from urllib.request import urlopen

import jax
from PIL import Image

import jaxnn

img = Image.open(urlopen(
    "https://huggingface.co/datasets/huggingface/cats-image/resolve/main/cats_image.jpeg"
))

model = jaxnn.create_model(
    "vit_base_patch16_224.sam_in1k",
    pretrained=True,
    num_classes=0,
)
model.eval()

data_config = jaxnn.data.resolve_model_data_config(model)
transforms = jaxnn.data.create_transform(**data_config, is_training=False)

x = jax.numpy.expand_dims(transforms(img), 0)
output = model(x, deterministic=True)

Citation

bibtex

@article{chen2021vision,
  title={When vision transformers outperform resnets without pre-training or strong data augmentations},
  author={Chen, Xiangning and Hsieh, Cho-Jui and Gong, Boqing},
  journal={arXiv preprint arXiv:2106.01548},
  year={2021}
}

bibtex

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}

bibtex

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}

JaxNN/vit_base_patch16_224.sam_in1k

Model Details

Model card for vit_base_patch16_224.sam_in1k

Model Details

Model Usage

Image Classification

Image Embeddings

Citation

JaxNN/
vit_base_patch16_224.sam_in1k