Views
No views yet
| Property | Value |
|---|---|
| Name | QiTianTokenizer-Medium |
| Type | Tokenizer-only repository |
| Purpose | General multilingual tokenization |
| Primary Languages | Chinese, English |
| Extended Support | Multilingual (Unicode-complete) |
| Architecture | Byte-level BPE |
| Vocabulary Size | 64,000 tokens |
| Fast Implementation | ✅ Available (QiTianTokenizerFast) |
| Framework | 🤗 transformers |
| License | Apache 2.0 |
| Variant | Vocabulary Size | Description | Recommended Use |
|---|---|---|---|
| QiTianTokenizer-Tiny | 12k | Lightweight tokenizer designed for compact or embedded models. | On-device or low-resource tasks |
| QiTianTokenizer-Base | 32k | Balanced vocabulary offering solid coverage and efficiency for most multilingual use cases. | Recommended for general use |
| QiTianTokenizer-Medium | 64k | Optimal balance in language coverage — broad enough to capture fine-grained linguistic diversity while maintaining reasonable model complexity. | Recommended for multilingual and high-quality general-purpose models |
| QiTianTokenizer-Large | 96k | Extended multilingual vocabulary designed for diverse cross-lingual pretraining and high-capacity language models. | High-resource training |
| QiTianTokenizer-XLarge | 128k | Full-script and domain-extensive vocabulary for comprehensive multilingual modeling. | Research & large-scale pretraining |
All variants share consistent token definitions, special tokens, and compatible configurations.
AutoTokenizer:1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("Morton-Li/QiTianTokenizer-Medium", trust_remote_code=True)
4
5# Example
6text = "你好，QiTian！"
7tokens = tokenizer(text)
8print(tokens["input_ids"])1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("Morton-Li/QiTianTokenizer-Medium", trust_remote_code=True)
4
5# Example
6texts = ["Hello, 世界！", "QiTian is multilingual."]
7batch_tokens = tokenizer(texts, padding=True, return_tensors="pt")
8print(batch_tokens["input_ids"])apply_chat_template)apply_chat_template:1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("Morton-Li/QiTianTokenizer-Medium", trust_remote_code=True)
4
5messages = [
6 {"role": "system", "content": "You are a helpful assistant."},
7 {"role": "user", "content": "你好，介绍一下 QiTianTokenizer。"},
8]
9
10text = tokenizer.apply_chat_template(
11 messages,
12 tokenize=False,
13 add_generation_prompt=True,
14 enable_thinking=False,
15)
16print(text)
17
18# If you need token ids directly:
19inputs = tokenizer.apply_chat_template(
20 messages,
21 tokenize=True,
22 add_generation_prompt=True,
23 enable_thinking=False,
24 return_tensors="pt",
25)
26print(inputs["input_ids"])add_generation_promptTrue: append the assistant role token (e.g. <|assistant|>) at the end, so the model can continue generating.False: do not append generation prompt (useful for evaluating full dialogues).enable_thinkingTrue: wrap the assistant part with a thinking span (e.g. <|begin_of_think|> ... <|end_of_think|>), if your training/inference uses it.False: keep plain assistant content without the thinking wrapper.| File | Description |
|---|---|
tokenizer.json | Serialized fast tokenizer definition |
tokenizer_config.json | Configuration (max length, padding side, etc.) |
tokenizer.py | Tokenizer implementation |
| Token | Purpose |
|---|---|
<|bos|> | Beginning of sequence |
<|eos|> | End of sequence |
<|eot|> | End of turn (marks message boundary) |
<|pad|> | Padding token for batch alignment |
<|mask|> | Masked token for MLM-style objectives |
<|system|> | Defines system or meta-instruction context |
<|user|> | Marks user message boundary in conversational data |
<|assistant|> | Marks assistant message boundary |
<|begin_of_think|> | Begin internal reasoning span |
<|end_of_think|> | End internal reasoning span |
1@misc{QiTianTokenizer,
2 title = {QiTianTokenizer: A Universal Multilingual Tokenizer with Chinese–English Optimization},
3 author = {Morton Li},
4 year = {2026},
5}