A Unigram (SentencePiece-style) tokenizer covering all three major Kurdish varieties in a single vocabulary: Kurmancî (Latin script), Soranî (Arabic script), and Zazakî (Latin script).

Trained by Kurdish-Tech on KurdishCorpus-clean, and measured on held-out text the tokenizer never saw during training.

Which variant should I use?

Four Kurdish tokenizers were trained and benchmarked together. This one has the best fertility on Soranî and Zazakî of all four — if those dialects matter most for your use case, start here. For Kurmancî-heaviest workloads, or for standard causal LM pretraining (Llama/GPT/Mistral-style architectures expect BPE), see kurdish-tokenizer-bpe-64k instead — it's marginally better on Kurmancî and is the more conventional choice for that family of models.

Fertility (tokens per word — lower is better)

Measured on held-out documents: 300 per dialect (166 for Zazakî — all that met the threshold), each ≥20 words, truncated to 2,000 characters. Every tokenizer below was measured with the same script on the same documents, encoding without special tokens.

Tokenizer	Kurmancî	Soranî	Zazakî
kurdish-bpe-64k	1.342	1.793	2.408
kurdish-unigram-64k (this model)	1.385	1.633	2.290
kurdish-bpe-32k	1.427	1.974	2.701
kurdish-unigram-32k	1.472	1.843	2.580
NLLB-200 (`distilled-600M`)	1.930	2.336	2.548
XLM-RoBERTa (`base`)	1.751	3.695	2.527
`o200k_base` (GPT-4o)	2.361	3.984	2.732
`cl100k_base` (GPT-4)	2.610	6.938	3.038

Against cl100k_base, this tokenizer needs 1.9× fewer tokens for Kurmancî and 4.2× fewer for Soranî — the best Soranî result of anything tested, including NLLB-200, which was purpose-built for 200 languages including Kurdish.

Usage

python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurdish-tech/kurdish-tokenizer-unigram-64k")

tok("Ez kurd im, ji Kurdistanê me.")          # Kurmancî
tok("زمانی کوردی زمانێکی دەوڵەمەندە.")          # Soranî
tok("Ma kirmanc î, zon u kulturê ma.")        # Zazakî

<bos> / <eos> are appended automatically. Like other SentencePiece/Unigram tokenizers, a leading space is part of the normalized representation — decode() reproduces the original text once that convention is accounted for (standard tokenizer.decode() handles this correctly; it only shows up if you inspect raw token strings).


Vocabulary size	64,000
Algorithm	Unigram (SentencePiece)
Special tokens	`<bos>`, `<eos>`, `<unk>`, `<pad>`, `<mask>`
`model_max_length`	1024

Training data

Same 800k-line sample as the BPE variants, over-weighted toward Soranî and Zazakî relative to their share of the corpus:

Dialect	Lines sampled	Share of training sample
Kurmancî	520,000	68.3%
Soranî	224,000	29.4%
Zazakî	17,309	2.3%

Other variants

kurdish-tokenizer-bpe-64k — best Kurmancî fertility, standard choice for causal LM
kurdish-tokenizer-bpe-32k — smaller vocab, smaller embedding table
kurdish-tokenizer-unigram-32k — smaller vocab, smaller embedding table

Limitations

Fertility is a relative comparison on a held-out sample, not an exhaustive evaluation. The Zazakî figure rests on 166 documents and is the least robust of the three.
Lower fertility means fewer tokens for the same text; it does not by itself guarantee better downstream model quality.
This is a tokenizer only — no language model weights are released here.

License & citation

Released under CC BY-SA 4.0, matching the corpus it was trained on.

bibtex

@misc{kurdishtech2026tokenizer,
  title  = {Kurdish Tokenizer (Unigram 64k): a multi-dialect tokenizer for Kurmanc\^i, Soran\^i and Zazak\^i},
  author = {{Kurdish-Tech}},
  year   = {2026},
  url    = {https://huggingface.co/kurdish-tech/kurdish-tokenizer-unigram-64k}
}

Built by Kurdish-Tech — open-source digital infrastructure for the Kurdish language.

Maintained by Alan Hesen.