This repository hosts the SentencePiece tokenizer model for the Odia language, created to support the efficient tokenization of Odia text in NLP applications. The tokenizer was built using a diverse dataset of Odia text, ensuring comprehensive language coverage and accurate tokenization.
Model Details
Model Prefix: odia_tokenizers_test
Model Type: BPE (Byte-Pair Encoding)
Vocabulary Size: 50,000 tokens
File Structure
odia_tokenizers_test.model: SentencePiece tokenizer model file.
odia_tokenizers_test.vocab: Vocabulary file containing all token mappings.
Installation and Usage
To load and use this tokenizer model, make sure you have the sentencepiece package installed:
pip install sentencepiece
python
1import sentencepiece as spm
2from huggingface_hub import hf_hub_download
34# Download the model file from Hugging Face5model_path = hf_hub_download(repo_id="shantipriya/OdiaTokenizer", filename="odia_tokenizers_test.model")67# Load the tokenizer model8sp = spm.SentencePieceProcessor()9sp.load(model_path)1011# Sample text for tokenization12text ="ଦୀପାବଳି ଏକ ଭାରତୀୟ ପର୍ବ ।"1314# Tokenize the text into pieces (subwords or tokens)15tokens = sp.encode_as_pieces(text)1617# Tokenize the text into token IDs (integer representations of the tokens)18token_ids = sp.encode_as_ids(text)1920# Print the tokenized output21print("Tokens:", tokens)22print("Token IDs:", token_ids)
Sample Tokenization
The model has been specifically trained on a diverse corpus of Odia text, ensuring high-quality tokenization results. Here’s an example of how the model tokenizes Odia sentences:
The vocabulary size was chosen to balance memory efficiency with language coverage, making it suitable for applications ranging from language modeling to text classification.
Vocabulary Statistics
Total Tokens: 50,000
Average Token Length: 6.46
Max Token Length: 16
Min Token Length: 1
Training and Configuration Details
The tokenizer was trained using the SentencePiece library with the following configurations:
Character Coverage: 99.995%
Input Sentence Size: 200 million sentences
Maximum Sentence Length: 4192 characters
Model Training Parameters:
shuffle_input_sentence=True
split_by_unicode_script=True
split_by_whitespace=True
byte_fallback=True
Intended Use
This model is intended for use in various NLP applications involving the Odia language, such as:
Language Modeling
Text Classification
Named Entity Recognition (NER)
Translation tasks involving Odia
License
This model is released under the cc-by-nc-sa-4.0 License.
Acknowledgments
This model was developed as part of a project to support low-resource language processing.
Thanks to OdiaGenAI for providing the initial training data, which made this model possible.
