Two independent SentencePiece tokenizers trained from scratch for a pair of monolingual ~25M-parameter decoder-only language models. They share no training text, no vocabulary and no merges — the separation is what makes a Hindi/Nepali comparison meaningful.

Each was selected from a sweep of BPE and Unigram × 16,000 and 24,000, then retrained on the complete cleaned corpus. tokenizer_sweep.json in each language folder holds the full comparison.

Fertility improves monotonically with vocabulary, so it cannot be the only criterion. At d_model=384 the embedding table costs vocab × 384 out of a ~25M budget, and one decoder layer costs ~12 × 384² = 1.77M:

A vocabulary much above ~32k spends the model on its own embedding table.

byte_fallback=True: any character the vocabulary cannot cover decomposes into UTF-8 byte tokens (ids 3–258). Nothing is ever discarded.

These were trained on text normalized by the project's common/normalize.py, whose behaviour is language-specific: chandrabindu (ँ) folds to anusvara (ं) for Hindi but not for Nepali, where the nasalization is phonemic.

Feeding un-normalized text to the Hindi model triggers byte fallback and inflates the token count. Always normalize with the matching language setting before encoding.

hindi/   hindi_final.model · hindi_final.vocab
         tokenizer_final.json · tokenizer_sweep.json · sweep_models/
nepali/  nepali_final.model · nepali_final.vocab
         tokenizer_final.json · tokenizer_sweep.json · sweep_models/