Supernova V3 is a scratch-built tokenizer infrastructure that achieves 7x the efficiency and speed of generic tokenizers (like Tiktoken or HuggingFace Fast Tokenizers) for the Nepali language.
Why to use this model?
Speed: 1.37x+ faster than older one on standard CPUs.
Power: 7x more efficient character-to-token representation for Devanagari.then older
No GPU Needed: Designed for high-speed inference on edge devices and standard
How to use
# Load the tokenizer_v3.json and use the encode/decode logic provided in the Supernova class.
Status: Shipped. The era of GPU-dependency for Nepali NLP ends here.
Supernova Nepali Tokenizer V3
A lightweight, CPU-oriented Nepali tokenizer using a custom Trie-based Max-Match Priority Encoder.
