The RoBERTa model was originally introduced in this paper. This model is a version of RoBERTa-small-Bulgarian fine-tuned for part-of-speech tagging.

The model can be used to predict part-of-speech tags in Bulgarian text. Since the tokenizer uses byte-pair encoding, each word in the text may be split into more than one token. When predicting POS-tags, the last token from each word can be used. Using the last token was found to slightly outperform predictions based on the first token.

An example of this can be found here.

The pretraining data is unfiltered text from the internet and may contain all sorts of biases.

In addition to the pretraining data used in RoBERTa-base-Bulgarian, the model was trained on the UPOS tags from (UD_Bulgarian-BTB)[https://github.com/UniversalDependencies/UD_Bulgarian-BTB].

The model was trained for 5 epochs over the training set. The loss was calculated based on label predictions for the last POS-tag for each word. The model achieves 98% on the test set.