This a tokenizer trained using
microtok and this is just a demo tokenizer for my blog which you can read
here--- TEST RESULTS ---
2026-03-19 08:58:41,165 - microtok.TikToken.tok - INFO - Input: 'Hello world! I am writing code today.'
2026-03-19 08:58:41,167 - microtok.TikToken.tok - INFO - Tokens: ['Hello', 'Ġworld', '!', 'ĠI', 'Ġam', 'Ġwriting', 'Ġcode', 'Ġtoday', '.']
1from transformers import PreTrainedTokenizerFast
2
3tokenizer = PreTrainedTokenizerFast.from_pretrained("Parveshiiii/microtok")
4
5print(f"Vocab size: {tokenizer.vocab_size}")
6print(tokenizer.tokenize("Hello from Panipat!"))
