You can deploy this model and start earning money today!

sidskarki/
deepseek-v4-nepali-tokenizer

Name: deepseek-v4-nepali-tokenizer
Availability: InStock

Views

No views yet

DeepSeek V4 — Nepali Extended Tokenizer

Extended tokenizer for DeepSeek V4 with ~15K added high-value Nepali/Devanagari tokens.

Token Efficiency

	Nepali tok/word
Original DeepSeek V4	4.01
Extended (this)	2.52
Reduction	37.3%

How It Was Built

Trained a 32K SentencePiece BPE tokenizer on a 7.49GB cleaned Nepali corpus
Selected tokens that the DeepSeek V4 base tokenizer splits into 3+ subtokens (delta vocabulary approach)
Added ~15K high-value Nepali tokens to the base tokenizer

The extended tokenizer is a drop-in replacement for the original. To use the new tokens effectively, the model needs continued pretraining on Nepali text (see the Qwen3-4B Nepali model for a full CPT+SFT example).

Usage

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sidskarki/deepseek-v4-nepali-tokenizer")
tokens = tokenizer.tokenize("नेपालको राजधानी काठमाडौं हो")
print(tokens, len(tokens))

Context

Part of a 17-model Nepali tokenizer benchmark measuring the Nepali token tax across modern LLM tokenizers.

sidskarki/deepseek-v4-nepali-tokenizer