This repository hosts the OTK-BPE family of production-grade Byte-Level BPE
(BBPE) tokenizers for regional and multilingual language models. It now
covers Swahili, Kinyarwanda, and a merged French + Kinyarwanda +
English + Swahili vocabulary — each available at three vocab sizes: 50k,
100k, and 150k — so you can pick the tradeoff between compactness and
coverage that fits your use case.
(Companion repo: olaverse/otk-bpe-50k
covers the Nigerian-languages family — Yoruba, Igbo, Hausa, Pidgin, and a
unified Naija tokenizer, all fixed at 50k.)
Which one should I use?
Your situation
Use
Only need Swahili
sw-150k (best fertility and entity handling in this family; drop to sw-50k/sw-100k only if embedding-table size is tightly constrained)
Only need Kinyarwanda
kin-150k
Need French, Kinyarwanda, English, and Swahili in one model
merged-150k
Storage/embedding-table size is a hard constraint
Step down to -100k or -50k in the same language — fertility degrades gradually, not a cliff (see Benchmarks)
150k is the recommended default across all three families — in every
benchmark below, fertility and entity-handling both improved monotonically
from 50k → 100k → 150k, with no exceptions. There's no size in this range
where a smaller vocab wins outright.
Key Features
Byte-Level BPE (BBPE): maps raw UTF-8 bytes to printable characters —
0.00% Out-Of-Vocabulary, zero [UNK] tokens, by construction.
Diacritic Preservation & Normalization: NFC normalization inside the
pre-tokenization chain, so accents and combining diacritics don't get split
into decomposed code points.
Code-Mixed English Support: the Swahili and Kinyarwanda tokenizers each
blend a English-Wikipedia component into training; the merged tokenizer
treats English as a full first-class language rather than a minority blend.
Emoji & Symbol Merging: curated emoji vocab is injected during training
so common emoji merge into single tokens instead of fragmenting.
Tokenizer Models in This Repository
All loaded from olaverse/otk-bpe, selected via the subfolder argument.
Subfolder
Languages
Vocab Size
sw-50k / sw-100k / sw-150k
Swahili
50,000 / 100,000 / 150,000
kin-50k / kin-100k / kin-150k
Kinyarwanda
50,000 / 100,000 / 150,000
merged-50k / merged-100k / merged-150k
French + Kinyarwanda + English + Swahili
50,000 / 100,000 / 150,000
Performance Benchmarks
Fertility = average tokens per word (lower is better). Entity fragmentation
= share of capitalized/likely-proper-noun words split into more than one
token (lower is better) — this metric predicts downstream NER difficulty
before any model is even trained.
Swahili benchmark uses the MasakhaNEWS Swahili test split (42,494
entity-candidate words checked) — a real, curated held-out set, never seen
during training.
Kinyarwanda, French, and English benchmarks use a held-out, non-training
slice of the same streaming sources (FineWeb-2 / Wikipedia) rather than a
curated test set — MasakhaNEWS does not include Kinyarwanda (only the
related-but-distinct Rundi/Kirundi), so no equivalent curated benchmark
exists for that language yet. Treat these three languages' numbers as
directionally reliable, not as rigorously verified as the Swahili figures.
Swahili
Tokenizer
Vocab Size
Fertility
Entity Fragmentation
otk-bpe sw-150k
150,000
1.210 (Best!)
0.341 (Best!)
otk-bpe sw-100k
100,000
1.233
0.435
AfroXLMR
250,002
1.597
0.441
otk-bpe merged-150k
150,000
1.264
0.519
otk-bpe sw-50k
50,000
1.285
0.626
otk-bpe merged-100k
100,000
1.302
0.672
mBERT
~119,547
2.071
0.575
otk-bpe merged-50k
50,000
1.393
0.822
GPT-4o (o200k_base)
200,019
1.841
0.768
GPT-4 (cl100k_base)
100,277
2.462
0.820
Kinyarwanda
Tokenizer
Vocab Size
Fertility
Entity Fragmentation
otk-bpe kin-150k
150,000
1.377 (Best!)
0.334 (Best!)
otk-bpe kin-100k
100,000
1.409
0.420
otk-bpe merged-150k
150,000
1.465
0.475
otk-bpe kin-50k
50,000
1.483
0.561
otk-bpe merged-100k
100,000
1.523
0.567
otk-bpe merged-50k
50,000
1.662
0.709
AfroXLMR
250,002
2.495
0.718
mBERT
~119,547
2.702
0.715
GPT-4o (o200k_base)
200,019
2.189
0.798
GPT-4 (cl100k_base)
100,277
2.798
0.854
Kinyarwanda is where general-purpose multilingual tokenizers are weakest
across the board — even the smallest dedicated tokenizer here (kin-50k)
beats every baseline on fertility, and kin-150k beats every baseline on
both metrics simultaneously.
French
Tokenizer
Vocab Size
Fertility
Entity Fragmentation
otk-bpe merged-150k
150,000
1.378 (Best!)
0.428
otk-bpe merged-100k
100,000
1.409
0.502
GPT-4o (o200k_base)
200,019
1.478
0.479
otk-bpe merged-50k
50,000
1.486
0.620
AfroXLMR
250,002
1.597
0.434
mBERT
~119,547
1.616
0.388 (Best!)
GPT-4 (cl100k_base)
100,277
1.720
0.578
English
Tokenizer
Vocab Size
Fertility
Entity Fragmentation
mBERT
~119,547
1.429 (Best!)
0.285 (Best!)
GPT-4o (o200k_base)
200,019
1.436
0.452
otk-bpe merged-150k
150,000
1.443
0.558
GPT-4 (cl100k_base)
100,277
1.460
0.496
otk-bpe merged-100k
100,000
1.500
0.659
AfroXLMR
250,002
1.538
0.425
otk-bpe merged-50k
50,000
1.632
0.803
English is the merged tokenizer's weakest relative showing — expected, since
none of the 4 languages get a dedicated vocabulary in the merged design, and
mBERT/AfroXLMR both have far larger, English-rich vocabularies to draw on.
merged-150k still lands within ~1% of mBERT's fertility despite covering 3
additional languages in the same vocabulary.
Reading the entity fragmentation numbers honestly
At 150k vocab, OTK-BPE tokenizers win entity fragmentation outright, not
just fertility, for Swahili and Kinyarwanda — sw-150k and kin-150k both
have the lowest fragmentation rate of every tokenizer tested, dedicated or
general-purpose. The vocab-size increase from 50k to 150k didn't just narrow
the gap to larger multilingual tokenizers on this metric, it closed and then
reversed it for these two languages specifically.
The exceptions are French and English, where mBERT — not AfroXLMR — has
the best entity handling (0.388 and 0.285 respectively), edging out even the
merged tokenizer. This makes sense: mBERT's vocabulary, while smaller than
AfroXLMR's, was built with heavy exposure to major European-language
Wikipedia text including large volumes of proper nouns, and English/French
are exactly where that shows. The merged tokenizer's own English and French
entity handling (0.558 and 0.428) is competitive but not best-in-class for
those two languages specifically — a real, honest tradeoff of covering 4
languages in one shared vocabulary rather than dedicating full budget to
either.
Net read: don't assume general-purpose tokenizers automatically win on
entity handling just because they have larger vocabularies — that held at
the smaller 50k vocab size tested earlier in this project, but stopped
holding once vocab size reached 150k for the languages that needed it most.
1from transformers import AutoTokenizer
23# Load the Swahili tokenizer at your chosen size4tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="sw-150k")56text ="Habari yako? Leo ni siku nzuri sana 😊"7inputs = tokenizer(text)89print("Tokens:", tokenizer.tokenize(text))10print("IDs:", inputs["input_ids"])11print("Decoded:", tokenizer.decode(inputs["input_ids"]))
