In the original Vision Transformers (ViT) paper (Dosovitskiy et al.), the authors concluded that to perform on par with Convolutional Neural Networks (CNNs), ViTs need to be pre-trained on larger datasets. The larger the better. This is mainly due to the lack of inductive biases in the ViT architecture -- unlike CNNs, they don't have layers that exploit locality.
Many groups have proposed different ways to deal with the problem of data-intensiveness of ViT training. One such way was shown in the Data-efficient image Transformers, (DeiT) paper (Touvron et al.). The authors introduced a distillation technique that is specific to transformer-based vision models. DeiT is among the first works to show that it's possible to train ViTs well without using larger datasets.
The model is trained for demonstrative purposes and does not guarantee the best results in production.
For better results, follow & optimize the
Keras example as per your need.
The model is trained and evaluated on
TF Flowers datasetTraining procedure is followed exactly as from the
keras example.
The batch size is however decreased to 16 from the original 256 for accomodating the model in a single V100 GPU memory.
