Standard BERT tokenizers (like bert-base-uncased) often struggle with Nigerian linguistic nuances. They tend to break down local words into meaningless sub-tokens (e.g., "Owanbe" might become "Ow", "##an", "##be").
NairaBERT Tokenizer was trained to recognize these as high-frequency units, ensuring that the model preserves the semantic meaning of Nigerian-centric text.
Base: Fine-tuned from bert-base-uncased logic but retrained on local corpora.
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("sijirama/nairabert-tokenizer")
4
5text = "the lagos traffic today was very bad, no cap."
6tokens = tokenizer.tokenize(text)
7print(tokens)
This tokenizer was trained on a scraped corpus of approximately 15M tokens. While it is a "beta" and proof-of-concept, it significantly reduces the sequence length for Nigerian text compared to standard Western-centric tokenizers (i think) .
