You can deploy this model and start earning money today!

HPLT/
hplt_bert_base_swh-Latn

Name: hplt_bert_base_swh-Latn
Availability: InStock

Views

No views yet

HPLT v2.0 for Swahili

This is one of the encoder-only monolingual language models trained as a second release by the HPLT project. It is a so called masked language model. In particular, we used the modification of the classic BERT model named LTG-BERT.

We present monolingual LTG-BERT models for more than 50 languages out of 191 total in the HPLT v2.0 dataset.

All the HPLT encoder-only models use the same hyper-parameters, roughly following the BERT-base setup:

hidden size: 768
attention heads: 12
layers: 12
vocabulary size: 32768

Every model uses its own tokenizer trained on language-specific HPLT data.

The training code.

The training statistics of all runs

Example usage

This model currently needs a custom wrapper from modeling_ltgbert.py, you should therefore load the model with trust_remote_code=True.

python

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("HPLT/hplt_bert_base_2_0_swh-Latn")
model = AutoModelForMaskedLM.from_pretrained("HPLT/hplt_bert_base_2_0_swh-Latn", trust_remote_code=True)

mask_id = tokenizer.convert_tokens_to_ids("[MASK]")
input_text = tokenizer("It's a beautiful[MASK].", return_tensors="pt")
output_p = model(**input_text)
output_text = torch.where(input_text.input_ids == mask_id, output_p.logits.argmax(-1), input_text.input_ids)

# should output: '[CLS] It's a beautiful place.[SEP]'
print(tokenizer.decode(output_text[0].tolist()))

The following classes are currently implemented: AutoModel, AutoModelMaskedLM, AutoModelForSequenceClassification, AutoModelForTokenClassification, AutoModelForQuestionAnswering and AutoModeltForMultipleChoice.

Intermediate checkpoints

We are releasing 10 intermediate checkpoints for each model at intervals of every 3125 training steps in separate branches. The naming convention is stepXXX: for example, step18750.

You can load a specific model revision with transformers using the argument revision:

model = AutoModelForMaskedLM.from_pretrained("HPLT/hplt_bert_base_2_0_swh-Latn", revision="step21875", trust_remote_code=True)

You can access all the revisions for the models with the following code: