This repository contains the weights for Jumbo, a simple and scalable architecture that makes Vision Transformers (ViTs) faster. Jumbo reduces patch token width while increasing global token width through a new "Jumbo" token processed by a shared, wider FFN.

ViTs are general and accurate, but often slow. Jumbo addresses this by reducing patch token width while adding a wider Jumbo token processed by its own wider FFN. This approach increases model capacity efficiently: the Jumbo FFN processes only a single token for speed, and its parameters are shared across all layers for memory efficiency. Crucially, Jumbo is attention-only and non-hierarchical, maintaining compatibility with plain ViT methods.

The following accuracies were achieved on ImageNet-1K:

For installation and running ImageNet-1K evals, attention visualization, and speed measurement, please follow the instructions in the official repository.