CIFAR-10 Results
In our CIFAR-10 experiment, we aimed to assess the performance of Mice ViTs on a more complex and diverse dataset, CIFAR-10. We trained mice ViTs on classifying the 10 classes.
Training Details
For each attention layer setting, we explored two model variants: an attention-only model and a model combining attention with the MLP module. Dropout and layer normalization were not applied for simplicity.
| Size | NumLayers | Attention Only | Attention+MLP |
|---|
| tiny | 1 | 0.43 | 0.479 |
| base | 2 | 0.47 | 0.54 |
| small | 3 | 0.47 | 0.51 |
| medium | 4 | 0.47 | 0.52 |
