A purpose-built 16K BPE tokenizer for the
toke programming language, achieving
52% average token reduction vs cl100k_base across 42 benchmark programs.
The toke code generation model (
karwalski/toke) uses Qwen's 151K vocab tokenizer internally. This tokenizer measures how efficiently toke code
could be tokenized by a future toke-native model.
1from tokenizers import Tokenizer
2
3tok = Tokenizer.from_file("tokenizer_v03.json")
4code = 'm=fib;f=fib(n:i64):i64{if(n<2){<n};<fib(n-1)+fib(n-2)};'
5result = tok.encode(code)
6print(f"{len(result.ids)} tokens") # 19 tokens (vs 49 cl100k)
Try the tokenizer in your browser at
tokelang.dev/tokenizer — see token boundaries highlighted with colours, side-by-side with cl100k.
