Finnish ModernBERT tiny-short-cpt is an encoder model following the ModernBERT architecture, pretrained on Finnish, Swedish, English, Code, Latin, and Northern Sámi. It was trained on 357.3B tokens. This is a last checkpoint before the LR decay phase, and we recommend using this model for continued pretraining. Training was conducted on the LUMI supercomputer. The project aimed to train multilingual encoder models that support long context and all official Finnish languages¹.

¹Multiple Sámi languages are spoken in Finland, but Northern Sámi is the most widespread and thus included in the training data. English is not the official language of Finland, but it is widely used. Latin was included for potential clinical use.

Full descriptions of training, data and evaluation are available in the article.

The Finnish ModernBERT models were competitive with other multilingual models (XLM-R-large and mmBERT-base) on short-context NLU tasks in Finnish, Swedish, and English, where XLM-R-large was the strongest model. The Finnish ModernBERTs are the strongest multilingual encoder models in out-of-domain retrieval tasks, outperforming the others with a large margin.

We used text datasets from diverse sources, including web crawls, news, scientific articles, classical literature, historical texts, Wikipedia, forums, and authoritative sources. Sources underwent various levels of pre-processing, including the removal of low-quality text or boilerplate, PII removal, and deduplication. Note that more datasets were used for the training than listed in this repository metadata.

Pretraining was done using Distributed Data Parallelism, AdamW with ZeroRedundancyOptimizer, and the WSD learning rate schedule. We describe the training as a three-step process where:

Finnish ModernBERTs' training data include sources regarded as biased and harmful, and the models' outputs may mirror these biased properties. The training data were not filtered for toxic, harmful, or offensive content to serve various use cases. The representations produced by the models should not be used without caution and without evaluating their effects on vulnerable population groups.

Finnish ModernBERT tiny-short-cpt is released under the Apache 2.0 license.

We acknowledge CSC, IT Center for Science, Finland, for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium. We acknowledge the HPLT-project for supporting this research. This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101070350, and it has also received funding from the Finnish Cultural Foundation. We thank The Language Bank of Finland for additional resources for Finnish, Finland-Swedish, and Swedish.

If you use Finnish ModernBERTs or need to reference the work, please use the citation below:

@misc{reunamo2025pretrainingfinnishmodernberts,
      title={Pretraining Finnish ModernBERTs}, 
      author={Akseli Reunamo and Laura-Maria Peltonen and Hans Moen and Sampo Pyysalo},
      year={2025},
      eprint={2511.09213},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.09213}, 
}