A linguistically-aware tokenizer for Tamil that decomposes words into their
root morphemes and grammatical suffixes instead of arbitrary byte-pair subwords.
Tamil is an agglutinative language — a single word can encode root + tense +
person + case + number + politeness, all fused together:
Surface form
Gloss
படிக்கிறான்
padik (study) + kiṟāṉ (3sg.m present)
படித்தாள்
padik + tāḷ (3sg.f past)
படிப்பார்கள்
padik + pār̤kaḷ (3pl future)
A BPE tokenizer sees three completely different token sequences for the same verb.
vettu always yields படி as the first token — the model learns shared semantics
across all inflections from the very first training step.
Benchmark: NanoGPT trained on Tamil Wikipedia
Same model (4-layer GPT, 128-dim, 128 context), same corpus, same steps:
Metric
BPE (4k vocab)
vettu
Vocab size
4,000
8,420
Best val loss
5.70
4.62
Perplexity
299
★ 102
Root consistency
58%
100%
Fertility (tokens/word)
2.96
1.65
vettu reaches 3× lower perplexity on the same data.
Generated text comparison
Prompt: தமிழ்நாட்டில்
BPE:, . ஆ ரம் , உ ற் பாலை வன த்துறை மற்றும் உ த்தி
→ Fragmented sub-character pieces, no linguistic meaning.
vettu:சென்னை இல் மாநிலம் இன் மாநில சென்னை ஐ மாவட்டம் இல் உள்ள து
→ Proper Tamil words with case markers (இல், இன், ஐ).
vettu is used for full morphological analysis of unseen words.
The pre-built word cache covers 15,000 most frequent Tamil Wikipedia words (~97% of
corpus tokens) — these work without vettu installed at runtime.
Coverage: top-15,000 most frequent Tamil Wikipedia words (~97% of tokens) in the pre-built word cache; vettu handles unseen words via rule-based morphology
