This t5 eff model has 728M parameters.
It was pre-trained with masked language modeling (denoise token span corruption) objective on the dataset
mc4_nl_cleaned config large_en_nl for 1 epoch(s) and a duration of 17d15h,
with a sequence length of 512, batch size 512 and 212963 total steps (56B tokens).
Pre-training evaluation loss and accuracy are 1,05 and 0,76.
Refer to the evaluation section below for a comparison of the pre-trained models on summarization and translation.
Pre-trained T5 models need to be finetuned before they can be used for downstream tasks, therefore the inference widget on the right has been turned off.
For a demo of the Dutch CNN summarization models, head over to the Hugging Face Spaces for
the Netherformer 📰 example application!
Please refer to the original T5 papers and Scale Efficiently papers for more information about the T5 architecture
and configs, though it must be noted that this model (t5-base-36L-dutch-english-cased) is unrelated to these projects and not an 'official' checkpoint.
The model uses a cased SentencePiece tokenizer configured with the Nmt, NFKC, Replace multi-space to single-space normalizers
and has 32003 tokens.
It was trained on Dutch and English with scripts from the Huggingface Transformers Flax examples.
See ./raw/main/tokenizer.json for details.
Dataset(s)
All models listed below are pre-trained on
cleaned Dutch mC4,
which is the original mC4, except
Sentences with a word of more than 1000 characters are removed
Documents with less than 5 sentences are removed
Documents with "javascript", "lorum ipsum", "terms of use", "privacy policy", "cookie policy", "uses cookies",
"use of cookies", "use cookies", "elementen ontbreken", "deze printversie" are removed.
The Dutch and English models are pre-trained on a 50/50% mix of Dutch mC4 and English C4.
The translation models are fine-tuned on CCMatrix.
Dutch T5 Models
Three types of Dutch T5 models have been trained (blog).
t5-base-dutch is the only model with an original T5 config.
The other model types t5-v1.1 and t5-eff have gated-relu instead of relu as activation function,
and trained with a drop-out of 0.0 unless training would diverge (t5-v1.1-large-dutch-cased).
The T5-eff models are models that differ in their number of layers. The table will list
the several dimensions of these models. Not all t5-eff models are efficient, the best example being the inefficient
t5-xl-4L-dutch-english-cased.
Most models from the list above have been fine-tuned for summarization and translation.
The figure below shows the evaluation scores, where the x-axis shows the translation Bleu score (higher is better)
and y-axis the summarization Rouge1 translation score (higher is better).
Point size is proportional to the model size. Models with faster inference speed are green, slower inference speed is
plotted as bleu.
Evaluation T5 Dutch English
Evaluation was run on fine-tuned models trained with the following settings:
Summarization
Translation
Dataset
CNN Dailymail NL
CCMatrix en -> nl
#train samples
50K
50K
Optimizer
Adam
Adam
learning rate
0.001
0.0005
source length
1024
128
target length
142
128
label smoothing
0.05
0.1
#eval samples
1000
1000
Note that the amount of training data is limited to a fraction of the total dataset sizes, therefore the scores
below can only be used to compare the 'transfer-learning' strength. The fine-tuned checkpoints for this evaluation
are not saved, since they were trained for comparison of pre-trained models only.
The numbers for summarization are the Rouge scores on 1000 documents from the test split.
The models below have been evaluated for English to Dutch translation.
Note that the first four models are pre-trained on Dutch only. That they still perform adequate is probably because
the translation direction is English to Dutch.
The numbers reported are the Bleu scores on 1000 documents from the test split.
The models t5-small-24L-dutch-english and t5-base-36L-dutch-english have been fine-tuned for both language
directions on the first 25M samples from CCMatrix, giving a total of 50M training samples.
Evaluation is performed on out-of-sample CCMatrix and also on Tatoeba and Opus Books.
The _bp columns list the brevity penalty. The avg_bleu score is the bleu score
averaged over all three evaluation datasets. The best scores displayed in bold for both translation directions.
This project would not have been possible without compute generously provided by Google through the
TPU Research Cloud. The HuggingFace 🤗 ecosystem was instrumental in all parts
of the training. Weights & Biases made it possible to keep track of many training sessions
and orchestrate hyper-parameter sweeps with insightful visualizations.
The following repositories where helpful in setting up the TPU-VM,
and getting an idea what sensible hyper-parameters are for training gpt2 from scratch:
