A Unigram tokenizer for Mon (mnw), Burmese and English, with full byte fallback.
Mon mixes with Burmese constantly and English routinely, so all three are trained
on and measured separately. Anything else on the page (Thai, emoji, IPA, CJK)
round-trips through byte fallback rather than being lost.
Both produce identical token ids. The package's model_card.json carries the
numbers below in machine-readable form, and a test fails if they drift from what
the artifact does.
Measured
Vocabulary 64,256 · Unigram · trained on the train split of an 893,936-line /
85.8M-character corpus · scored on the whole validation split.
stratum
chars/token
tokens/line
syllable violations
round-trip
byte-fallback tokens
Mon
4.686
16.4
1.07% (n=492,469)
100%
20 / 486,631
Burmese
4.117
13.9
0.93% (n=25,546)
100%
0 / 24,942
English
4.112
34.6
— (n=0)
100%
96 / 453,350
mixed script
3.804
24.6
0.81% (n=28,133)
100%
116 / 59,440
98.74% of the distinct characters in the Mon validation split are single
tokens — 392 of 397, over all 29,600 lines. The remaining five (a combining
diaeresis below, a Greek capital pi, two Sinhala letters and one emoji; seven
occurrences in 2.28M characters) round-trip through byte fallback at one to four
tokens each.
Four notes on reading these:
Compression describes a corpus, not a tokenizer. Quote it with the corpus
or not at all. On a different distribution you will get a different number.
Byte fallback is given as a count, not a rate. Mon's 20 tokens in 486,631
round to a two-decimal zero, which reads as never; Burmese's zero is a real
zero. Those are different facts and a percentage renders them identically.
Syllable violations count token boundaries falling inside a Myanmar
syllable. They are measured with a purpose-built segmenter, not Unicode
grapheme clusters — UAX #29 puts a cluster break before ာ, so \X scores a
cut through the middle of ကျော် as clean. The denominator is given because
English legitimately has none.
Round-trip compares after normalization, which is a deliberate transform
and not a loss.
Everything round-trips
python
1for text in["🙏 emoji","ภาษาไทย","漢字","Ωπ√∫","ကျော် page 42 — “quoted” ၏"]:2assert tokenizer.decode(tokenizer.encode(text, add_special_tokens=False))== text
The vocabulary carries all 256 <0xNN> byte pieces, so nothing is unrepresentable.
This matters if you are feeding OCR output into a corpus: without byte fallback,
characters outside the vocabulary are not flagged — they are deleted, leaving
fluent-looking output with content silently missing.
Normalization travels with the model
The normalizer is stored insidetokenizer.json, so it cannot drift from the
weights: invisible characters stripped, Unicode space separators folded to
U+0020, then NFC. Runs of spaces are preserved.
That is a fix, not a feature. The previous release applied no normalization at
encode time while having been trained on normalized text, so a single ZERO
WIDTH SPACE cost five tokens instead of one.
Special tokens
token
id
<unk>
0
<s>
1
</s>
2
<pad>
3
All four have real ids. In the previous release <pad> was declared but never
assigned one, so padding a batch padded with <unk> — silently.
Upgrading from the previous release
Every token id has changed. This is a retrain on a new algorithmic stack, so
any embedding matrix built against the old vocabulary must be rebuilt. Pin
mon-tokenizer<1.0 if you are not ready.
Two things about the previous release are worth stating plainly, because they
were live:
tokenizer.json here was the 4,000-piece predecessor, not the 32,000-piece
model the card described. AutoTokenizer prefers tokenizer.json over
tokenizer.model, so anyone loading this repo through the documented path got
a tokenizer measuring 0.93 chars/token against an advertised 5.22. This
release replaces it; there is no tokenizer.model any more, so the two cannot
disagree again.
The advertised 5.22 chars/token did not reproduce on an independent corpus.
The numbers above were measured on held-out text and are stated with the corpus
they came from.
Why Unigram
Measured at equal vocabulary on this corpus: BPE compresses better everywhere
(+6.2% Mon, +25.8% English) but splits Myanmar syllables 2.5× more often.
Byte-level BPE is far behind: Myanmar is three UTF-8 bytes per character, giving
1.524 chars/token against 4.507, both at 48,000 pieces over the 29,600-line Mon
validation split.
Unigram wins for an OCR consumer, where the syllable is the unit a reader sees.
Full reasoning and the rejected alternatives:
docs/architecture.md.
MIT for the card and the configs. Not for tokenizer.json.
The vocabulary was fitted to a corpus this project neither collected nor owns, so
it is a derived work and its terms follow that corpus. The Mon portion comes from
MonCorpusCollection, whose
largest source is CC BY-SA 4.0 Mon Wikipedia. Its second largest, the Independent
Mon News Agency archive, has no established licence at all.
LICENSE states which sources are established and which are not.
Nothing here grants rights over tokenizer.json, because none were obtained.
The front-matter said mit until 2026-08-13, which claimed more than this
repository holds.
