Karpotron Tokenizer
Overview
Karpotron is a specialized tokenizer based on NVIDIA's Nemotron-3-Nano-30B tokenizer, optimized for Ukrainian language processing. It replaces 28,065 vocabulary slots with Ukrainian/Cyrillic BPE entries while maintaining the original 131,072 vocabulary size by pruning tokens from writing systems geographically and culturally distant from Ukraine.
Key Features
-
+28,065 Cyrillic BPE entries:
- 152 base Cyrillic letters
- 27,425 Ukrainian surface tokens from lapa-llm/tokenizer
- 488 reachable Ukrainian replacement/intermediate nodes used to repair strict BPE reachability
-
No removal of English or EU language tokens - only non-essential tokens from distant writing systems were replaced
-
Latin-safe removal - preserved Latin tokens with diacritics used in multiple European languages (ü, ö, ç, ã, á, é, etc.)
-
Identical specifications - vocab size (131,072), byte-level BPE encoding match original Nemotron
Reachability Repair
The tokenizer was repaired with a strict byte-level BPE reachability audit over the replaced ID range (id > 103006) and then checked against the actual tokenizers ByteLevel pre-tokenizer + BPE pipeline. The current artifact passes:
| Checked inserted entries | Atomic entries | Broken entries |
|---|
| 28,065 | 28,065 | 0 |
During repair, 488 previously unreachable or pipeline-unreachable surface slots were replaced with reachable Ukrainian replacement/intermediate nodes while keeping the vocabulary size and ID range unchanged. See reachability_repair_report.md and reachability_actual_tokenizers_report.md for examples.
Replaced Tokens by Writing System
The tokenizer replaced 24,360 tokens from 17 writing systems while adding 28,065 Ukrainian tokens:
| Writing System | Original | Removed | Retained | % Removed |
|---|
| Arabic | 9,400 | 8,458 | 942 | 90.0% |
| Hangul (Korean) | 4,492 | 3,330 | 1,162 | 74.1% |
| Han (Chinese) | 3,767 | 2,364 | 1,403 | 62.8% |
| Devanagari (Hindi) | 1,554 | 1,155 | 399 | 74.3% |
| Hebrew | 1,002 | 722 | 280 | 72.1% |
| Telugu | 920 | 675 | 245 | 73.4% |
| Bengali | 839 | 615 | 224 | 73.3% |
| Armenian | 1,121 | 549 | 572 | 49.0% |
| Thai | 567 | 423 | 144 | 74.6% |
| Kannada | 570 | 419 | 151 | 73.5% |
| Tamil | 539 | 394 | 145 | 73.1% |
| Malayalam | 406 | 283 | 123 | 69.7% |
| Georgian | 513 | 240 | 273 | 46.8% |
| Hiragana/Katakana (Japanese) | 1,623 | 208 | 1,415 | 12.8% |
| Gujarati | 204 | 136 | 68 | 66.7% |
| Gurmukhi | 155 | 111 | 44 | 71.6% |
| Myanmar | 234 | 96 | 138 | 41.0% |
Fully preserved:
- Latin scripts (English, Spanish, French, German, Italian, Portuguese, Dutch, Danish, Swedish, Polish)
- Greek (1,507 tokens, 100% retained)
Metrics
Acknowledgement: evaluation results provided by @Sofetory
| |lang-uk/malyuk [100k] |allenai/c4(en) [100k]|allenai/c4 (es,fr,it,de) [100k]|QIRIM/crh (Cyrillic) [94]|allenai/c4(ru) [100k]|allenai/c4(bg) [100k] |allenai/c4(be) [100k]|
|words count |22,898,164 |36,170,971 |198,173,216 |1,868,259 |42,557,519 |44,627,199 |43,153,645 |
| | | | | | | | |
|tokenizers |tokens |toks/word |tokens |toks/word |tokens |toks/word |tokens |toks/word |tokens |toks/word |tokens |toks/word |tokens |toks/word |
|Qwen/Qwen3-8B |84,408,084 |3.686 |46,884,593 |1.296 |395,581,536 |1.996 |7,956,741 |4.259 |116,115,062 |2.728 |132,597,427 |2.971 |173,571,099 |4.022 |
|meta-llama/Llama-3.1-8B-Instruct |57,226,997 |2.499 |46,085,724 |1.274 |382,143,751 |1.928 |7,386,873 |3.954 |104,974,733 |2.467 |119,123,733 |2.669 |150,189,294 |3.48 |
|microsoft/Phi-4-mini-instruct |59,447,036 |2.596 |45,423,925 |1.256 |335,188,687 |1.691 |5,995,822 |3.209 |91,824,464 |2.158 |102,472,523 |2.296 |119,587,038 |2.771 |
|CohereLabs/aya-expanse-8b |50,973,632 |2.226 |47,364,187 |1.309 |353,221,932 |1.782 |6,614,719 |3.541 |93,089,697 |2.187 |112,612,668 |2.523 |141,262,943 |3.273 |
|google/gemma-3-12b-it |57,388,402 |2.506 |47,285,432 |1.307 |354,241,840 |1.788 |6,240,944 |3.341 |95,520,817 |2.245 |103,950,626 |2.329 |131,398,147 |3.045 |
|nvidia/NVIDIA-Nemotron-3-Nano-30B|62,087,149 |2.711 |47,630,139 |1.317 |365,218,644 |1.843 |6,623,516 |3.545 |107,233,038 |2.520 |108,691,963 |2.436 |135,489,439 |3.140 |
|karpotron-tokenizer (Ours) |46,456,626 |2.029 |47,650,584 |1.317 |365,285,307 |1.843 |7,519,362 |4.025 |132,519,787 |3.114 |131,626,936 |2.949 |158,657,784 |3.677 |
|Comments |25% fewer tokens than Nemotron on Ukrainian|English unchanged|EU languages unchanged|QIRIM slightly worse|Russian drops (UA-centric)|Bulgarian drops slightly|Belarusian drops|
Usage Example
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained(
4 "transhumanist-already-exists/karpotron-tokenizer"
5)
6
7toks = tokenizer("Всі красиві зберігають оптимізм", add_special_tokens=False)
8print(toks.input_ids) # 6 tokens with the current reachability-repaired artifact
Model Contents
- tokenizer.json - Byte-level tokenizer spec (131,072 tokens, 258,350 merges)
- tokenizer_config.json - Configuration metadata
- special_tokens_map.json - Special token mappings (identical to Nemotron)
- reachability_repair_report.md/json - Strict BPE graph reachability repair summary
- reachability_actual_tokenizers_report.md/json - Actual
tokenizers pipeline reachability repair summary
Embedding Initialization
For newly added tokens in Nemotron models, you can:
- Use tools like Focus or Zett
- Initialize embeddings randomly with warm-up schedule training
- Unchanged tokens (103,007) retain original IDs and can reuse existing embeddings
Citation
1@misc{karpotron_tokenizer,
2 author = {Bohdan Didenko},
3 title = {Karpotron Tokenizer},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/transhumanist-already-exists/karpotron-tokenizer}},
6 note = {Based on NVIDIA Nemotron-3-Nano-30B and lapa-llm/tokenizer}
7}
Base Models
