A BERT model pretrained on Latvian language data using the masked language modeling and next sentence prediction objectives.
It was introduced in
this paper and first released via a
GitHub repository.
The current HF repository contains an improved version of LVBERT.
This model is case-sensitive. It is primarily intended to be fine-tuned on downstream natural language understanding tasks like text classification, named entity recognition, question answering.
However, the model can be used as is to compute contextual embeddings for tasks like text similarity and clustering, semantic search.
A SentencePiece model was trained on the training dataset, producing a vocabulary of 32,000 tokens.
It was then converted to the WordPiece format used by BERT.
We used the BERT-base configuration with 12 layers, 768 hidden units, 12 heads, 512 sequence length, 128 mini-batch size and 32k token vocabulary.
1@inproceedings{Znotins-Barzdins:2020:BalticHLT,
2 author = {Arturs Znotins and Guntis Barzdins},
3 title = {{LVBERT: Transformer-Based Model for Latvian Language Understanding}},
4 booktitle = {Human Language Technologies - The Baltic Perspective},
5 series = {Frontiers in Artificial Intelligence and Applications},
6 volume = {328},
7 publisher = {IOS Press},
8 year = {2020},
9 pages = {111-115},
10 doi = {10.3233/FAIA200610},
11 url = {http://ebooks.iospress.nl/volumearticle/55531}
12}
