A 32,000-token multilingual byte-level tokenizer covering the full FLORES-200 set (204 languages, 27+ scripts) with zero coverage failures. VI.1.3 reads as 6.1.3 in Quartz notation (the leading Roman V is the numeral 5, so the VI generation is 6). This is the Prelude tier: a 32K tokenizer tuned to keep English and the major languages competitive with industry 32K tokenizers while covering the entire long tail losslessly. The planned Overture tier (64K) is intended to raise the low-resource complex scripts without an English trade-off.
At the same 32K budget as Llama 2 and Mistral, QT VI.1.3 is a genuinely better multilingual tokenizer. It covers all 204 FLORES-200 languages losslessly, where Llama 2 and Mistral each fall back to bytes on 34; it beats Llama 2 on 107 of the 170 languages Llama can tokenize at all, and Mistral on 137; and it roughly doubles their efficiency on Arabic, Hindi and Thai while leading on CJK. The cost is a modest, honest tax on English and European text (about 8% behind Llama 2), the price of universal coverage at a fixed vocabulary.
What it is
Vocabulary: 32,000 tokens.
Algorithm: two-stage byte-level BPE. Stage 1 learns subwords; Stage 2 learns cross-word "superwords" under a BoundlessBPE sanity rule, so partial-word fragment merges (the "e t" junk class) are impossible by construction.
Coverage: all 204 FLORES-200 languages, 27+ scripts, no zero-coverage script.
Base: byte-level. Every byte sequence is representable, so there is no UNK token and decoding is exactly lossless.
Correctness guarantees
Lossless round-trip on all 204 languages: decode(encode(x)) == x byte-for-byte.
No UNK emission anywhere.
No byte-fallback collapse; no zero-coverage script.
FLORES-200 health gate: 0 correctness failures across all 204 languages.
Head-to-head vs industry 32K tokenizers (FLORES-200 devtest)
Coverage
tokenizer
vocab
languages covered
zero-coverage / byte-fallback failures
QT VI.1.3
32K
204 / 204
0
Llama 2
32K
170 / 204
34
Mistral
32K
170 / 204
34
Llama 2 and Mistral have no dedicated tokens for many scripts (for example CJK, Tibetan, Telugu, Myanmar, Armenian, Lao) and fall back to raw bytes, which the coverage check flags as failures.
Compression, chars per token (higher is better)
language
QT VI.1.3
Llama 2
Mistral
English
3.90
4.24
4.26
German
3.17
3.50
3.17
French
3.17
3.44
3.22
Italian
3.16
3.42
3.16
Spanish
3.33
3.48
3.26
Russian
2.43
2.82
2.51
Chinese
0.94
0.70
0.88
Japanese
1.25
0.82
0.87
Arabic
2.11
1.09
1.11
Hindi
2.12
0.92
0.95
Thai
1.80
0.93
0.99
QT trails Llama 2 by about 8% on English and the European languages, sits at or above Mistral on German, Italian and Spanish, and is far ahead of both on every non-Latin script. Across the 170 languages Llama 2 can tokenize, QT wins on 107; against Mistral it wins on 137.
vs the predecessor, QT V.4.6 (FLORES-200, chars per token, higher is better)
VI.1.3 fixes the English over-fragmentation that motivated this line. It improves on 98 of 204 languages, is more equitable overall (spread 6.4x vs 7.6x), and lifts English and the major languages sharply; the trade is on CJK and the low-resource complex scripts, which are the 64K Overture target.
language
QT VI.1.3
QT V.4.6
delta
English
3.90
3.11
+0.79
German
3.17
2.85
+0.32
French
3.17
2.90
+0.26
Arabic
2.11
1.92
+0.19
Spanish
3.33
3.20
+0.13
Russian
2.43
2.38
+0.05
Chinese
0.94
1.06
-0.11
Japanese
1.25
1.45
-0.20
Thai
1.80
2.34
-0.54
Tibetan
1.84
2.67
-0.84
English
English was the motivating case for this line. VI.1.3:
1.53 tokens per word on running English text, at 3.90 chars per token, which trails Llama 2's 4.24 by about 8%.
Common long words are single or paired tokens: committee and recommended take 2 tokens each, international and development a single token.
This is competitive with industry 32K tokenizers on English while retaining full multilingual coverage. Closing the final few percent to match Llama 2 exactly is planned for the Overture (64K) tier, where the larger budget is expected to remove the coverage-versus-efficiency trade.
Special tokens
The tokenizer reserves 86 special tokens (IDs 0-85) for downstream model use:
Core:<|padding|> (0), <|bos|> (1), <|endoftext|> (2, EOS), <|unk|> (3), <|sep|> (4). The <|unk|> token is declared for convention only; the byte-level base means it is never emitted.
Language control: 72 <|lang:xx|> tags (for example <|lang:en|>, <|lang:ja|>, <|lang:ar|>) for optional language steering.
These are reserved slots; no chat template is bundled, so a downstream model can define its own.
Usage
python
1from tokenizers import Tokenizer
23tok = Tokenizer.from_file("tokenizer.json")45enc = tok.encode("The committee recommended further investigation.")6print(enc.ids)7print(enc.tokens)89# Lossless round-trip across scripts10text ="中华人民共和国 / भारत / ประเทศไทย / العربية"11assert tok.decode(tok.encode(text).ids)== text
With Hugging Face Transformers:
python
1from transformers import AutoTokenizer
23tok = AutoTokenizer.from_pretrained("JamesQuartz/qt-VI.1.3-32k")4tok("The quick brown fox")
Files: tokenizer.json (self-contained), plus vocab.json and merges.txt for compatibility.
Version notation
Quartz uses a leading Roman V as the numeral 5. So V.4.6 reads as 5.4.6, and VI.1.3 reads as 6.1.3: the sixth-generation architecture (VI), first line (.1), third iteration (.3). Never written V6.1.3.
Intended use and limitations
A general-purpose multilingual tokenizer for pretraining and inference across the full FLORES-200 set, tuned so English and major languages stay competitive with industry 32K tokenizers while nothing in the long tail is left uncovered. At 32K it is a strong generalist rather than a per-language optimum; the lowest-resource complex scripts (for example Tibetan) are constrained by both the fixed budget and available training data, and are the target of the 64K Overture tier.
Citation
@misc{qt_vI_1_3_ultralingo,
title = {QT VI.1.3 UltraLingo: a coverage-complete 32K multilingual tokenizer},
author = {Quartz (AENEA Global)},
year = {2026},
howpublished = {\url{https://huggingface.co/JamesQuartz/qt-VI.1.3-32k}}
}
