Extended tokenizer for Qwen 3 with ~15K added high-value Nepali/Devanagari tokens.
The extended tokenizer is a drop-in replacement for the original. To use the new tokens effectively, the model needs continued pretraining on Nepali text (see the
Qwen3-4B Nepali model for a full CPT+SFT example).
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("sidskarki/qwen3-nepali-tokenizer")
4tokens = tokenizer.tokenize("नेपालको राजधानी काठमाडौं हो")
5print(tokens, len(tokens))
Part of a
17-model Nepali tokenizer benchmark measuring the Nepali token tax across modern LLM tokenizers.
