This repository hosts the new 1.42B Turkish T5 model named BERT5urk.
BERT5urk is part of the Turkish Model Zoo family and pretrained using the awesome
T5X library with the UL2 objective.
Inspired by the great Finnish T5 and UL2 models from the Finnish NLP
group, BERT5urk also uses UL2 and the efficient T5 architecture, that is proposed in the "Scale Efficiently" paper. Many thanks
to the Finnish NLP group for open-sourcing the pretraining code and models!
Pretraining Data
BERT5urk uses the Turkish part of the amazing FineWeb2 corpus.
Only documents with a higher language score than 0.99 are chosen for final pretraining corpus, that has a total size of 262GB.
We train a SPM-based vocab on a 3GB corpus from randomly chosen documents of the pretraining corpus.
Pretraining
BERT5urk was pretrained with the awesome T5X library. Some pretraining highlights:
One-shot pretraining (pretraining without any training crashes) was possible a v3-32 TPU Pod and took 16.56 days
Model was pretrained for 2M steps for an input & output sequence length of 512 and a batch size of 128
The resulting model has 1.42B parameters
Evaluation
Detailed evaluations can be found in the Turkish Model Zoo repository. Additionally, we also fine-tuned
TURNA models as it is another T5 model with 1.14B parameters for comparison.
Encoder-only Results
For experiments on named entity recognition (NER) and part-of-speech (PoS) tagging we also the awesome Flair library and fine-tune only the encoder of BERT5urk and TURNA.
The overall performance can be seen in the following table:
We tried to replicate the results from the TURNA paper using the TURNA fine-tuning library.
Paraphrasing - Tatoeba
We fine-tune five different models for both TURNA and BERT5urk with different seeds and report the average score. Additionally the score from the TURNA paper
is also shown in the following table:
