This is the Ultra Pro version designed for maximum quality, advanced compression vs 256k Ultra version for very large models , and precision in complex real-world applications.

JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b

Open Robot platform

Tiangong : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
Unitree g1 https://a.co/d/0e4A8YVc
LimX Oli https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
ubtrobot https://www.ubtrobot.com/en/
x-humanoid https://www.x-humanoid.com/detail/hskw.html

Key Features

Algorithm: Byte-Level BPE
Vocabulary Size: 524,288 tokens — excellent balance between precision and efficiency
Multilingual & Technical Strength: Optimized for English, Russian, code, scientific literature, and technical documentation
Domain Specialization: Strong performance on programming languages, engineering, robotics, and scientific texts

Special Tokens Support

Full ChatML dialogue format (<|im_start|>, <|im_end|>)
FIM (Fill-in-the-Middle) support for code generation
Rich set of domain routing tokens (__CODING__, __PYTHON__, __ROBOTICS__, __SCIENCE__, etc.)
Extended robotics and control tokens

CMS Manhattan Service & Support

Jirack patent guards your technology for competitors
Redesign Llama , Qwen , Gemma to Ternary model
Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347
Accelerate inference via high compression 512K tokenizer and replace multiplication with sum operations via Ternary weights

Usage

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-UltraPro-Tokenizer-512K")

text = "__CODING__ __PYTHON__ Write a merge sort function in Python."

tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("Token IDs:", token_ids)

Vocab size: 524288
pad_token_id: 2
eos_token_id: 1

JiRack Pretrain Dataset

https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

bash

python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]

Benchmark for tokens quality .

bash

=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 31576, 522, 66472, 6176, 2037, 576, 7637, 269, 4, 454]

=== Decoding Token by Token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
31576 -> ' Write'
522 -> ' a'
66472 -> ' merge'
6176 -> ' sort'
2037 -> ' function'
576 -> ' in'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== ChatML 模板处理后的文本 ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|>


=== Token (ID) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 196893, 7637, 476, 24410, 16482, 61950, 14333, 295880, 92252, 870, 4, 454]

=== 逐个 Token 解码 ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
196893 -> ' 用'
7637 -> ' Python'
476 -> ' '
24410 -> '写'
16482 -> '一个'
61950 -> '归'
14333 -> '并'
295880 -> '排序'
92252 -> '函数'
870 -> '。'
4 -> '<|im_end|>'
454 -> '
'


== Texte après le modèle ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 170797, 3433, 4484, 56203, 595, 3102, 34759, 720, 7637, 269, 4, 454]

=== Décodage token par token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
170797 -> ' Éc'
3433 -> 'ris'
4484 -> ' une'
56203 -> ' fonction'
595 -> ' de'
3102 -> ' tri'
34759 -> ' fusion'
720 -> ' en'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Text nach ChatML-Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|>


=== Token (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 115144, 18528, 6806, 256728, 268, 144869, 268, 386798, 592, 576, 7637, 269, 4, 454]

=== Dekodierung Token für Token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
115144 -> ' Schre'
18528 -> 'ibe'
6806 -> ' eine'
256728 -> ' Merge'
268 -> '-'
144869 -> 'Sort'
268 -> '-'
386798 -> 'Funkt'
592 -> 'ion'
576 -> ' in'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== النص بعد تطبيق قالب ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|>


=== الرموز (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 33139, 9120, 360514, 209685, 6513, 5867, 1187, 580, 121802, 6176, 264, 474504, 84721, 2518, 1435, 269, 4, 454]

=== فك الترميز رمزا برمز ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
33139 -> ' اك'
9120 -> 'تب'
360514 -> ' دالة'
209685 -> ' فرز'
6513 -> ' بال'
5867 -> 'دم'
1187 -> 'ج'
580 -> ' ('
121802 -> 'merge'
6176 -> ' sort'
264 -> ')'
474504 -> ' بلغة'
84721 -> ' باي'
2518 -> 'ث'
1435 -> 'ون'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Текст после ChatML шаблона ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|>


=== Токены (ID) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 35013, 91654, 492868, 430356, 174599, 302515, 467815, 874, 66017, 269, 4, 454]

=== Декодирование по токенам ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
35013 -> ' Нап'
91654 -> 'иши'
492868 -> ' функцию'
430356 -> ' сорти'
174599 -> 'ровки'
302515 -> ' слия'
467815 -> 'нием'
874 -> ' на'
66017 -> ' python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

Email: grabko@cmsmanhattan.com
Phone: +1 (516) 777-0945
Location: New York, USA

CMSManhattan/JiRack-UltraPro-Tokenizer-512K

JiRack Utra Pro Tokenizer 512