Views
No views yet
Why it matters: Fewer tokens = more usable context per input = better learning & longer text coverage.
##k, ##z, ##ab).nirapekza + jYAna).t → d, i → y).transformers:
Note: The model expects SLP1 transliteration (e.g., vidyA), not Devanagari.1from transformers import AutoTokenizer
2
3# Load with trust_remote_code=True because of custom logic
4tokenizer = AutoTokenizer.from_pretrained(
5 "ArthaLabs/panini-tokenizer",
6 trust_remote_code=True
7)
8
9# Tokenize complex Sandhi compounds (SLP1 input)
10text = "nirapekzajYAnasAkzAtkArasAmarthyam"
11tokens = tokenizer.tokenize(text)
12
13print(tokens)| Input Compound | Panini (Ours) | Google MuRIL | Qwen2 |
|---|---|---|---|
nirapekzajYAnasAkzAtkArasAmarthyam | 6 | 18 | 25 |
tadekaniScitArthavyavasthApanam | 6 | 13 | 18 |
svaprakASatvaparaprakASavyavacCedaH | 7 | 15 | 22 |
svAtantryAbhAvasamucchinnakartRtvanirAsaH | 8 | 19 | 25 |
▁nirapekza | jYAna | sAkzAtkAra | sAman | arthy | am (6 meaningful roots)nirape | ##k | ##z | ##a | ##jya | ##nas... (14 noise fragments)1@misc{panini2025,
2 author = {ArthaLabs},
3 title = {Panini Tokenizer: Grammar-First Sanskrit Tokenization},
4 year = {2025},
5 publisher = {Hugging Face},
6 howpublished = {\url{https://huggingface.co/ArthaLabs/panini-tokenizer}}
7}