| Metric | Value |
|---|---|
| Vocab Size | 65,536 |
| Myanmar Tokens | 56,341 (85.97%) |
| Compression | 0.1718 |
| Avg Token Length | 5.83 |
| Coverage | 54.18% |
| Bits/Byte | 2.5412 |
| Metric | Original Qwen | Myanmar | Improvement |
|---|---|---|---|
| Compression | 1.4607 | 0.1718 | +88.2% |
| Coverage | 3.11% | 54.18% | +1639.5% |
1from transformers import AutoTokenizer
2tokenizer = AutoTokenizer.from_pretrained("kkomyoeminaung/qwen3-14b-myanmar-tokenizer")
3tokens = tokenizer.tokenize("မြန်မာနိုင်ငံရဲ့ မြို့တော်က နေပြည်တော်ပါ။")
4print(tokens)