Pretrained multilingual language model using a masked language modeling (MLM) objective.

ModernBERT is a transformers model pretrained on 1.2 billions of multilingual Wikipedia and OPUS tokens in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts.

This model has the following configuration:

You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task.

Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at model like GPT ones.

Here is how to use this model to get the features of a given text in PyTorch:

The ModernBERT model was pretrained on 1.2 billion of token from Multilingual Wikipedia (excluding lists, tables and headers) and OPUS.

It have been extracted from a 3.2 billion token corpus using an entropy sampling method called "diversity sampling" (paper to come).

The texts are cased and tokenized using SentencePiece and a vocabulary size of 128,000 tokens plus 1,000 unused token for downstream adataption. The inputs of the model are then of the form:

The tools used to pre-train the model are available here