SentencePiece BPE tokenizers trained on a 7.49GB cleaned Nepali corpus for Devanagari-optimized tokenization.
Three vocab sizes included: 32K, 48K, and 64K.
For comparison, English baseline is ~1.25 tokens/word. These tokenizers bring Nepali close to English-level efficiency.
Assembled from 4 sources, cleaned with Unicode NFC normalization, Devanagari ratio filtering (>50%), control character removal, and paragraph-level deduplication:
Total: 7.49GB, 2.83B characters, 18.7M lines.
1import sentencepiece as spm
2
3sp = spm.SentencePieceProcessor()
4sp.load("nepali_bpe_32k.model")
5
6tokens = sp.encode("नेपालको राजधानी काठमाडौं हो", out_type=str)
7print(tokens)
Built as part of a
17-model Nepali tokenizer benchmark. High-value tokens from the 32K model were used to extend production model tokenizers (Phi-4, Qwen 3.5, DeepSeek V4, Kimi K2.6), reducing their Nepali token counts by 37-52%.
