This is a custom tokenizer trained specifically for the Sinhala language using Byte-Pair Encoding (BPE), intended to be used in GPT-style language models and NLP applications. The tokenizer was trained on a Sinhala dataset to provide high-quality tokenization for downstream tasks like language modeling, classification, or generation.
This tokenizer is designed for Sinhala language processing. It has been trained on a cleaned 250MB Sinhala corpus using the Hugging Face tokenizers library. The training strategy is optimized for use in GPT-style auto-regressive models, with ByteLevel pre-tokenization and decoding for better whitespace handling.
You can use this tokenizer directly for any Sinhala text tokenization or for training/fine-tuning a GPT model.
This tokenizer reflects the patterns of its training data. If your corpus contains biased, offensive, or skewed data, those patterns may be reflected in tokenization and downstream generation. Always evaluate on real-world examples.
1from transformers import PreTrainedTokenizerFast
2
3tokenizer = PreTrainedTokenizerFast.from_pretrained("Navanjana/sinhala-gpt-tokenizer")
4
5text = "ඔබට සුබ දවසක් වේවා"
6tokens = tokenizer.tokenize(text)
7print(tokens)
