rost-tok-ro-raw
Romanian BPE tokenizer trained on raw (mixed-encoding) text. Published for comparison; prefer the normalized one.
Measured fertility
| tokenizer | tokens/word on Romanian |
|---|
| English-only (control) | 2.8421 |
| Romanian, raw (mixed encodings) | 1.5660 |
| Romanian, normalized | 1.5571 |
| Romanian, diacritics stripped | 1.5190 — but 2.0083 on correct Romanian |
A Romanian tokenizer is worth 1.8× over an English one. Stripping
diacritics looks 2.5% better on de-diacriticised text and is 29% worse on
correct Romanian, because it byte-falls-back on every ș ț ă â î.
