Pretrained ConvBERT model on Finnish language using a replaced token detection (RTD) objective. ConvBERT was introduced in this paper and first released at this page.

Note: this model is the ConvBERT generator model intented to be used for the fill-mask task. The ConvBERT discriminator model intented to be used for fine-tuning on downstream tasks like text classification is released here Finnish-NLP/convbert-base-finnish

Finnish ConvBERT is a transformers model pretrained on a very large corpus of Finnish data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts.

More precisely, it was pretrained with the replaced token detection (RTD) objective. Instead of masking the input like in BERT's masked language modeling (MLM) objective, this approach corrupts the input by replacing some tokens with plausible alternatives sampled from a small generator model. Then, instead of training a model that predicts the original identities of the corrupted tokens, a discriminative model is trained that predicts whether each token in the corrupted input was replaced by a generator model's sample or not. Thus, this training approach resembles Generative Adversarial Nets (GAN).

This way, the model learns an inner representation of the Finnish language that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard classifier using the features produced by the ConvBERT model as inputs.

Compared to BERT and ELECTRA models, ConvBERT model utilizes a span-based dynamic convolution to replace some of the global self-attention heads for modeling local input sequence dependencies. These convolution heads, together with the rest of the self-attention heads, form a new mixed attention block that should be more efficient at both global and local context learning.

You can use this generator model mainly just for the fill-mask task. For other tasks, check the Finnish-NLP/convbert-base-finnish model instead.

Here is how to use this model directly with a pipeline for fill-mask task:

The training data used for this model contains a lot of unfiltered content from the internet, which is far from neutral. Therefore, the model can have biased predictions. This bias will also affect all fine-tuned versions of this model.

This Finnish ConvBERT model was pretrained on the combination of five datasets:

Raw datasets were cleaned to filter out bad quality and non-Finnish examples. Together these cleaned datasets were around 84GB of text.

The texts are tokenized using WordPiece and a vocabulary size of 50265. The inputs are sequences of 512 consecutive tokens. Texts are not lower cased so this model is case-sensitive: it makes a difference between finnish and Finnish.

The model was trained on TPUv3-8 VM, sponsored by the Google TPU Research Cloud, for 1M steps. The optimizer used was a AdamW with learning rate 1e-4, learning rate warmup for 20000 steps and linear decay of the learning rate after.

Training code was from the official ConvBERT repository and also some instructions was used from here.

For evaluation results, check the Finnish-NLP/convbert-base-finnish model repository instead.

This project would not have been possible without compute generously provided by Google through the TPU Research Cloud.

Feel free to contact us for more details 🤗