Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:

More details can be found in our GitHub repository.

We use the official JAX/FLAX example in Hugging Face Transformers to pretrain a ByT5 model on a single v3-8 TPU. Details about the training can be found here.

We evaluated the hmByT5 model on ICDAR Europeana dataset:

We only performed fine-tuning for one epoch. Unfortunately, this ByT5 Base model shows no improvement over ByT5 Small architecture.

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️