Views
No views yet
huggingface/tokenizers on the cleaned Kazakh corpus
Abzalbek89/corpus_clean (~1.5M documents).| Metric | Value |
|---|---|
| Rank | 8 / 13 (lower fertility is better) |
| Vocabulary size | 32,000 |
| Fertility (tokens / word) | 2.346 |
| Compression (chars / token) | 3.345 |
| Compression (bytes / token) | 6.056 |
| Total tokens / words | 6,958,922 / 2,966,207 |
Abzalbek89/corpus_clean (14,831+ documents, ~3M words). Full ranking and methodology: Abzalbek89/kk-tokenizer-fertility-baseline.Abzalbek89/kk-tokenizer-fertility-baseline. Without that step the tokenizer will still encode the input but the morpheme-aware merge bias is lost.1import morfessor
2from huggingface_hub import hf_hub_download
3from transformers import AutoTokenizer
4
5tok = AutoTokenizer.from_pretrained('Abzalbek89/kk-tokenizer-morph-hf-unigram-32k')
6morf_path = hf_hub_download(
7 'Abzalbek89/kk-tokenizer-fertility-baseline', 'morfessor.bin', repo_type='dataset')
8io = morfessor.MorfessorIO()
9model = io.read_binary_model_file(morf_path)
10
11def morph_segment(text):
12 out = []
13 for w in text.lower().split():
14 morphemes, _ = model.viterbi_segment(w)
15 out.extend(morphemes)
16 return ' '.join(out)
17
18ids = tok.encode(morph_segment('Қазақстан Республикасы'), add_special_tokens=False)1from transformers import AutoTokenizer
2
3tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-tokenizer-morph-hf-unigram-32k")
4ids = tok.encode("Қазақстан Республикасының мемлекеттік тілі қазақ тілі.",
5 add_special_tokens=False)
6print(len(ids), tok.tokenize("Қазақстан Республикасының мемлекеттік тілі қазақ тілі."))Abzalbek89/corpus_cleanAbzalbek89/kk-tokenizer-fertility-baseline1@misc{kk_tokenizer_2026,
2 title = {Tokenizer Optimization for Kazakh Small Language Models},
3 author = {Abzalbek Ulasbek},
4 year = {2026},
5 howpublished = {Hugging Face Hub: \url{https://huggingface.co/Abzalbek89/kk-tokenizer-morph-hf-unigram-32k}},
6}