Linguistically-correct tokenization for six Brahmic (Indic) scripts. The tokenizer
treats the Akshara (orthographic syllable) as the atomic writing unit rather
than the byte. It has two layers: a rule-based Akshara segmenter (pure Unicode
rules, no model) that never splits an Akshara, and a SentencePiece model trained on
the segmented stream. The Python library that provides the segmenter and the
high-level tokenizer lives on GitHub and PyPI, linked below.
In v1.2 each Akshara the map covers is replaced by a single private-use codepoint
before SentencePiece, so no token boundary can fall inside a covered Akshara.
decode(encode(text)) is byte-identical, reconstructed from the id stream alone
with no side channel. This was verified on a 20-string probe set covering the six
scripts, mixed script, nukta forms, conjuncts, and whitespace edges (20 of 20), and
on 1,750 FLORES-200 devtest lines, 250 each from the six Indic scripts and English
(1,750 of 1,750). v1.1 could not round-trip at all, because its pipeline space-joins
aksharas and the model normalizes whitespace. 45,274 of the 64,000 v1.2 vocabulary
pieces span two or more aksharas.
Fertility
FLORES-200 devtest, tokens per whitespace-delimited word (lower is better). Numbers
match the GitHub README (benchmark_2026_07/results_v1_2.md).
Script (language)
v1.2 (64k)
v1.1 (16k)
Qwen3-14B
Devanagari (Hindi)
1.374
2.451
4.757
Gurmukhi (Punjabi)
1.445
2.684
7.758
Tamil (Tamil)
1.953
5.007
10.064
Telugu (Telugu)
2.001
3.710
11.406
Bengali (Bengali)
1.797
3.384
7.117
Kannada (Kannada)
2.058
4.166
11.876
Overall (six Indic)
1.717
3.411
8.398
English (info)
2.151
5.084
1.261
v1.2 roughly halves v1.1's token count on every Indic script and stays far below
Qwen3-14B. On English, Qwen3 is more efficient, as expected for an English-centric
byte-level BPE.
Against Indic-specific tokenizers
Qwen3-14B is a general multilingual BPE and a weak baseline for an Indic claim. Against
tokenizers built for Indian languages, on the same FLORES-200 devtest: budget-matched
against sarvam-1 (68,096 vocab, closest to v1.2's 64,000), v1.2 uses fewer tokens on all
six native scripts, by 2.00 to 14.15 percent. Against sarvam-30b (262,144 vocab, 4.1x the
budget) it wins five of six, losing Bengali by 6.76 percent.
That comparison is not like for like. sarvam and Krutrim are full LLM tokenizers carrying
English, code and multilingual coverage in one vocabulary, while v1.2 spends 92.25 percent
of its 64,000 slots on Indic aksharas. A native-script win partly reflects that
specialisation, not engineering alone. The classification rule behind that 92.25 percent is
stated in full under "Vocabulary classifier" in results_competitor_comparison.md, so the
figure is checkable rather than asserted. An earlier revision of this card said 91.94
percent; that figure could not be reproduced and is superseded.
Round-trip differs too. v1.2 is 1,012 of 1,012 on all six scripts; Qwen3-14B ranges from
403 to 1,011. That is not data loss: Qwen3 applies Unicode NFC normalization inside its
tokenizer, so its output is canonically equivalent to the input and renders identically
without being byte-identical. It is a reasonable design choice. The tradeoff is
canonicalization against exact reconstruction.
Where this tokenizer is the wrong choice
On Latin-script input the advantage inverts, by design rather than by defect: with 92.25
percent of the vocabulary spent on Indic aksharas, only 2,251 slots (3.52 percent) go to
Latin pieces, at a mean length of 3.44 characters.
On Dakshina human romanizations v1.2 uses more tokens than Qwen3-14B, by 23.4 percent on
romanized Hindi, 16.0 on Punjabi and 12.8 on Tamil. On code-switched Hinglish it is last
of five at 2.377 tokens per word, against 1.448 for sarvam-30b at 262,144 vocab.
If your workload is predominantly romanized or code-switched Latin text, a large
general-purpose tokenizer will serve you better. If it is native script, this one will not.
Full tables and method in benchmark_2026_07/results_competitor_comparison.md on GitHub.
That file also reports tokens per 100 characters alongside tokens per word, for
cross-script comparison. It preserves the tokenizer-to-tokenizer ratios within each script,
so it changes no conclusion above.
Akshara split rate
FLORES-200 devtest, percent of aksharas whose token boundaries fall inside them
(lower is better).
Script
v1.2 (64k)
v1.1 (16k)
Devanagari
0.0945
1.6906
Gurmukhi
0.3345
0.9348
Tamil
0.0000
0.1467
Telugu
0.0643
3.6511
Bengali
0.0721
1.7465
Kannada
0.1154
3.6202
Overall
0.1109
1.8559
For v1.2, a covered akshara is one codepoint, so the residual is only the
coverage-trim tail, not vocabulary pressure. v1.2 byte-fallback is near zero on all
six scripts (0.00 to 0.04 percent).
The mapping table is required
The v1.2 ids are not decodable without the mapping table. Each piece is built from
private-use codepoints, and the map (akshara_tokenizer_v1_2.map.json) translates
those back to aksharas. The map ships next to the model and is bound to it by
sha256, so loading refuses a mismatched pair. Always fetch the model and its map
together, and do not store ids without both.
Versions and compatibility
v1.1 and v1.2 produce different, incompatible id streams. Nothing in the API records
which model produced a stream, so decoding v1.1 ids with the v1.2 model returns
silent garbage rather than an error. If you already have a corpus tokenized with
v1.1, keep using v1.1 (the files remain in this repository, and PyPI 1.1.0
installs it) rather than upgrading in place.
Precomposed nukta letters (Gurmukhi and Bengali)
Precomposed nukta letters encode to 2 or 3 pieces rather than 1, because the map holds
their decomposed equivalents: Gurmukhi U+0A36 as U+0A38 followed by U+0A3C, Bengali
U+09DF as U+09AF followed by U+09BC. This costs tokens, never correctness: these aksharas
carry zero byte-fallback and round-trip remains byte-identical. On FLORES-200 it affects
0.33 percent of Gurmukhi aksharas.
Gurmukhi and Bengali pay the same penalty, and NFC-normalizing the input recovers it:
Script
as is
NFC
change
Gurmukhi
1.4446
1.3619
-5.72 percent
Bengali
1.7973
1.6912
-5.90 percent
Round-trip is unaffected by normalization form: 1,012 of 1,012 lines in all eighteen
cells, six scripts across as-is, NFC and NFD input.
NFD input costs tokens instead, up to 33.61 percent on Kannada, but sarvam-1 degrades
37.91 percent on the same input, so that is a general Brahmic-NFD effect rather than
something specific to this tokenizer.
The NFC recommendation is a measured conclusion, not an untested workaround. Rebalancing the
training corpus toward the precomposed forms was tested at three levels, without touching
the map. It recovers at most 41.7 percent of the Gurmukhi penalty and 17.3 percent of the
Bengali penalty, while degrading other scripts, because within a fixed corpus every
occurrence one form gains the other loses. The penalty is therefore structural within a
fixed vocabulary budget, and it is the price of byte-exact round-trip. Round-trip held at
1,012 of 1,012 in all eighteen cells throughout.
Detail in benchmark_2026_07/results_bengali_diagnostic.md and
benchmark_2026_07/results_arm_n_normalization.md on GitHub.
Usage (v1.2, fetching files from this repo)
Install the library (it provides the segmenter and the AksharaTokenizer class),
then download both v1.2 files and load them together:
The simpler path is PyPI: pip install "akshara-tokenizer[model]" already bundles
the v1.2 model and map, so AksharaTokenizer.load() works with no download.
IYRA is a separate project by the same author. IYRA does not use this tokenizer: it
runs on the Qwen3-14B tokenizer. AksharaTokenizer is developed independently and
shares no runtime code with IYRA. NyayaBench is the benchmark used to evaluate IYRA
and is likewise unrelated to this tokenizer.
