Rust fast decode corrupts some added tokens when a code point’s low byte is ≤ 32 (e.g. č → \r). See tokenizers#1996 and the upstream fix in tokenizers#1995.

This repo ships ZlatorogTokenizerFast (tokenization_zlatorog.py), which decodes added tokens the same way as the Transformers 4.x slow Zlatorog tokenizer. Token ids and vocabulary strings are unchanged.

Requirements

transformers>=4.45 or >=5.0
tokenizers>=0.22
trust_remote_code=True (loads ZlatorogTokenizerFast)

Usage

python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained(
    "zidsi/Zlatorog-30B-MoE-tokenizer",
    trust_remote_code=True,
)

word = "Začnimo"
ids = tok.encode(word, add_special_tokens=False)
assert tok.decode(ids) == word

Use this tokenizer with zidsi/Zlatorog-30B-MoE-CPT_Long (or any checkpoint trained with the same vocabulary).

Audit

321 of 25 893 added tokens were affected on the Hub revision audited for the parent model (a2759ee7565dc7c55c9c93c3f9e72190dcf5def4). See the companion repo’s artifacts/affected_added_tokens.json for the full checklist.