Enjoy — We extend the JiRack Models Ecosystem! 🚀
- The Best RAG with Mixture-of-Experts via CMS Manhattan RAG System with JiRack Ternary Experts for CPU Solution.
- JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b
JiRack Router Tokenizer Pro
JiRack Pro Tokenizer - 347 active language editions of Wikipedia
High-performance custom Byte-Level BPE tokenizer trained on the full Wikipedia dump across multiple languages.
Developed specifically for intelligent routing models and Mixture-of-Experts systems.
Model Details
- Developer: Konstantin Grabko (JiRack)
- License: Apache License 2.0
- Training Data: Full Wikipedia multilingual dump
- Vocabulary Size: 65,536 tokens
- Special Tokens: 128 reserved tokens (including 40+ domain routing tokens)
Key Features
- Correctly placed
<|unk|> token at ID 0
- Full native support for ChatML format (
<|im_start|> / <|im_end|>)
- Large set of specialized routing tokens (
__CODING__, __MATH__, __PYTHON__, __SCIENCE__, etc.)
- Support JiRack Robotics Technology with tags (
<|action_start|> / <|action_end|>)
- Support JiRack vision , images , audio-visual
- Strong multilingual performance
Basic system and dialogue tokens
"<|unk|>",
"<|endoftext|>",
"<|padding|>",
"<|im_start|>",
"<|im_end|>",
Core roles
"<|im_start|>system",
"<|im_start|>user",
"<|im_start|>assistant",
Additional useful tokens
"<|im_start|>tool",
"<|im_start|>function",
Reasoning block
"<|im_start|>thought",
Tool calls
"<|tool_call|>",
"<|tool_response|>",
Multimodality and audio-visual block
"<|image|>",
"<|video|>",
"<|sound|>",
"<|voice|>",
"<|listening|>",
"<|vision|>",
Emotional state (Mood)
"<|mood_happy|>",
"<|mood_sad|>",
"<|mood_angry|>",
"<|mood_neutral|>",
--- FIM (Fill-in-the-Middle) tokens from StarCoder ---
"<|fim_prefix|>",
"<|fim_middle|>",
"<|fim_suffix|>",
Robotics (trajectory/command boundaries)
"<|action_start|>",
"<|action_end|>",
Highlights
- Architecture: Byte-Level BPE (Byte-level Byte Pair Encoding) which natively prevents Out-Of-Vocabulary (OOV) tokens.
- Form Factor: Fully wrapped into Hugging Face
PreTrainedTokenizerFast with native ByteLevel decoders for clean cyrillic representation.
- Chat Standard: Out-of-the-box support for ChatML formatting (
<|im_start|> / <|im_end|>).
- Domain Specialization: Pre-baked atomic routing tokens like
__CODING__ and __PYTHON__ etc.
CMS Manhattan Service & Support
- Jirack patent guards your technology for competitors
- Redesign Llama , Qwen , Gemma to Ternary model
- Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347
- Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights
Usage
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Router-Tokenizer-65K")
4
5text = "<|im_start|>user\n__CODING__ __PYTHON__ Write a merge sort function.<|im_end|>"
6print(tokenizer.encode(text))
JiRack Pretrain Dataset
1python train_jirack_accelerate.py
2Processing jirack_pretrain_chunk_0.pt: 27%|█████████████████████▋ | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
3Processing jirack_pretrain_chunk_0.pt: 87%|███████████████████████████████████████████████████████████████████████ | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]
Benchmark for tokens quality .
1=== Text after ChatML Template ===
2<|im_start|>system
3You are a precise router model.<|im_end|>
4<|im_start|>user
5__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>
6
7
8=== Tokens (IDs) ===
9[5, 326, 5095, 944, 396, 23348, 1021, 7831, 5869, 141, 4, 326, 6, 326, 29, 348, 44, 26876, 396, 52698, 6521, 2031, 460, 7524, 141, 4, 326]
10
11=== Decoding Token by Token ===
12[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
135 -> '<|im_start|>system'
14326 -> '
15'
165095 -> 'You'
17944 -> ' are'
18396 -> ' a'
1923348 -> ' precise'
201021 -> ' ro'
217831 -> 'uter'
225869 -> ' model'
23141 -> '.'
244 -> '<|im_end|>'
25326 -> '
26'
276 -> '<|im_start|>user'
28326 -> '
29'
3029 -> '__CODING__'
31348 -> ' '
3244 -> '__PYTHON__'
3326876 -> ' Write'
34396 -> ' a'
3552698 -> ' merge'
366521 -> ' sort'
372031 -> ' function'
38460 -> ' in'
397524 -> ' Python'
40141 -> '.'
414 -> '<|im_end|>'
42326 -> '
43'
📧 Contact & Licensing
For joint ventures, hardware integration, or licensing inquiries:
- Email: grabko@cmsmanhattan.com
- Phone: +1 (516) 777-0945
- Location: New York, USA
📧 Copyright 2026 CMS Manhattan . All rights reserved
