Views
No views yet
['ह', 'र', 'à¥ĩ', 'Ġà¤ķ', 'à¥', 'ĥ', 'ष', 'à¥įà¤', '£'] (36 tokens)['▁हरे', '▁कृष्ण', '▁हरे', '▁कृष्ण', '▁कृष्ण', '▁कृष्ण', '▁हरे', '▁हरे'] (8 tokens)1from transformers import AutoTokenizer
2
3# Load tokenizer (native Hugging Face format)
4tokenizer = AutoTokenizer.from_pretrained("diabolic6045/Sanskrit-English-qwen2-tokenizer")
5
6# Test Sanskrit tokenization
7text = "हरे कृष्ण हरे कृष्ण कृष्ण कृष्ण हरे हरे"
8tokens = tokenizer.tokenize(text)
9print(tokens) # ['▁हरे', '▁कृष्ण', '▁हरे', '▁कृष्ण', '▁कृष्ण', '▁कृष्ण', '▁हरे', '▁हरे']
10
11# Perfect reconstruction
12decoded = tokenizer.decode(tokenizer.encode(text))
13print(decoded) # "हरे कृष्ण हरे कृष्ण कृष्ण कृष्ण हरे हरे"
14
15# Chat template support
16messages = [{'role': 'user', 'content': 'What is the meaning of हरे कृष्ण?'}]
17formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
18print(formatted)| Tokenizer | Tokens | Readable | Efficiency | Format |
|---|---|---|---|---|
| Ours | 8 | YES | 4.5x better | Native HF |
| Qwen | 36 | NO | Poor | ByteLevel BPE |
1# qwen.yaml
2base_model: Qwen/Qwen2.5-1.5B
3tokenizer_config: diabolic6045/Sanskrit-English-qwen2-tokenizer
4resize_token_embeddings_to_32x: true1# Start training
2accelerate launch -m axolotl.cli.train qwen.yaml