Views
No views yet
| Language Group | Accuracy |
|---|---|
| English/European | 95-100% |
| Korean | 70% |
| Japanese | 81% |
| Chinese | 7% (still learning) |
| Rare Languages | 47% avg |
1from transformers import AutoModel, AutoTokenizer
2
3# Load model
4model = AutoModel.from_pretrained("woo-jinhyun/intelligent-tokenizer-v6")
5tokenizer = ByteTokenizerV6() # Custom tokenizer
6
7# Process text
8text = "안녕하세요"
9encoded = tokenizer.encode(text)
10compressed = model.encode(encoded)1@software{intelligent_tokenizer_2025,
2 author = {Woo, Jinhyun and Claude Code},
3 title = {Intelligent Tokenizer: Language Pattern Learning},
4 year = {2025},
5 url = {https://github.com/Woojiggun/intelligent-tokenizer}
6}