The training dataset was tokenized into subwords using HerBERT Tokenizer; a character level byte-pair encoding with
a vocabulary size of 50k tokens. The tokenizer itself was trained on Wolne Lektury and a publicly available subset of
National Corpus of Polish with a fastBPE library.
Tokenizer utilizes XLMTokenizer implementation for that reason, one should load it as allegro/herbert-klej-cased-tokenizer-v1.
HerBERT models summary
Model
WWM
Cased
Tokenizer
Vocab Size
Batch Size
Train Steps
herbert-klej-cased-v1
YES
YES
BPE
50K
570
180k
Model evaluation
HerBERT was evaluated on the KLEJ benchmark, publicly available set of nine evaluation tasks for the Polish language understanding.
It had the best average performance and obtained the best results for three of them.
If you use this model, please cite the following paper:
@inproceedings{rybak-etal-2020-klej,
title = "{KLEJ}: Comprehensive Benchmark for {P}olish Language Understanding",
author = "Rybak, Piotr and
Mroczkowski, Robert and
Tracz, Janusz and
Gawlik, Ireneusz",
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
month = jul,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.acl-main.111",
doi = "10.18653/v1/2020.acl-main.111",
pages = "1191--1201",
}
Authors
The model was trained by Allegro Machine Learning Research team.
