A byte-level BPE tokenizer covering all three major Kurdish varieties in a single vocabulary: Kurmancî (Latin script), Soranî (Arabic script), and Zazakî (Latin script).

Trained by Kurdish-Tech on KurdishCorpus-clean, and measured on held-out text the tokenizer never saw during training.

Which variant should I use?

Honestly, probably not this one. Four Kurdish tokenizers were trained and benchmarked together, and this 32k-vocabulary BPE variant has the weakest fertility of all four on every single dialect. Its only reason to exist is a smaller embedding table (32k vs 64k rows) for parameter-constrained models — e.g. targeting a small on-device model where every embedding row has a real memory cost. If that constraint doesn't apply to you, use kurdish-tokenizer-bpe-64k instead — same algorithm, meaningfully better fertility on every dialect, for a vocabulary that's still small by modern standards.

Fertility (tokens per word — lower is better)

Measured on held-out documents: 300 per dialect (166 for Zazakî — all that met the threshold), each ≥20 words, truncated to 2,000 characters. Every tokenizer below was measured with the same script on the same documents, encoding without special tokens.

Tokenizer	Kurmancî	Soranî	Zazakî
kurdish-bpe-64k	1.342	1.793	2.408
kurdish-unigram-64k	1.385	1.633	2.290
kurdish-bpe-32k (this model)	1.427	1.974	2.701
kurdish-unigram-32k	1.472	1.843	2.580
NLLB-200 (`distilled-600M`)	1.930	2.336	2.548
XLM-RoBERTa (`base`)	1.751	3.695	2.527
`o200k_base` (GPT-4o)	2.361	3.984	2.732
`cl100k_base` (GPT-4)	2.610	6.938	3.038

Still beats cl100k_base on every dialect (1.8× fewer tokens on Kurmancî, 3.5× fewer on Soranî) despite being the weakest of the four Kurdish variants.

Usage

python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurdish-tech/kurdish-tokenizer-bpe-32k")

tok("Ez kurd im, ji Kurdistanê me.")          # Kurmancî
tok("زمانی کوردی زمانێکی دەوڵەمەندە.")          # Soranî
tok("Ma kirmanc î, zon u kulturê ma.")        # Zazakî


Vocabulary size	32,000
Algorithm	Byte-level BPE
Special tokens	`<bos>`, `<eos>`, `<unk>`, `<pad>`, `<mask>`
`model_max_length`	1024

Other variants

kurdish-tokenizer-bpe-64k — same algorithm, better fertility (recommended)
kurdish-tokenizer-unigram-64k — best Soranî/Zazakî fertility of all four
kurdish-tokenizer-unigram-32k — Unigram at the same vocab size

Limitations

Fertility is a relative comparison on a held-out sample, not an exhaustive evaluation.
This is the weakest-fertility variant of the four Kurdish tokenizers published here — published for completeness and for use cases genuinely constrained on vocabulary size, not as the general recommendation.
This is a tokenizer only — no language model weights are released here.

License & citation

Released under CC BY-SA 4.0, matching the corpus it was trained on.

bibtex

@misc{kurdishtech2026tokenizer,
  title  = {Kurdish Tokenizer (BPE 32k): a multi-dialect tokenizer for Kurmanc\^i, Soran\^i and Zazak\^i},
  author = {{Kurdish-Tech}},
  year   = {2026},
  url    = {https://huggingface.co/kurdish-tech/kurdish-tokenizer-bpe-32k}
}

Built by Kurdish-Tech — open-source digital infrastructure for the Kurdish language.

Maintained by Alan Hesen.