Views
No views yet
tokenizer.json)tokenizer.model + a custom ChatGLMTokenizer(PreTrainedTokenizer)) — there is no fast
tokenizer.json in the upstream repo. The Rust engine's tokenizer loader (mlx_gen::TextTokenizer,
consumed by the Kolors generator and the Kolors LoRA/LoKr trainer) reads the HF tokenizers fast
serialization, so it needs a tokenizer.json.tokenizer.json so SceneWorks model-install can overlay it onto the
upstream Kolors-diffusers snapshot (instead of running a Python SentencePiece→fast conversion at
install time on every machine — a Python-eradication consideration, epic 3482).tools/build_kolors_tokenizer.py (mlx-gen): converts the ChatGLM3 SP model to a fast
tokenizer.json via transformers' SP converter. The fast tokenizer reproduces the SP content ids
exactly; it adds no special tokens — the ChatGLM [gMASK] (64790) / sop (64792) prefix, left-pad,
and position_ids are applied by the Rust KolorsTokenizer wrapper (matching
build_inputs_with_special_tokens + _pad, max_length=256).sp_model.encode(text) across an EN + EN-long(truncation)[gMASK]=64790, sop=64792, pad=unk=0).tokenizer.json — the derived fast tokenizer (the file the Rust engine needs).tokenizer.model — the upstream ChatGLM3 SentencePiece model (provenance / reproducibility).tokenizer_config.json — the upstream tokenizer config.