PersianBPETokenizer Model Card

Model Details

Model Description

Comparing Performance on a pragraph of persian text

Model Type

• Tokenization Algorithm: Byte-Pair Encoding (BPE)
• Normalization: NFD, StripAccents, Lowercase, Strip, Replace (ZWNJ)
• Pre-tokenization: Whitespace
• Post-processing: TemplateProcessing for special tokens

Model Version

• Version: 1.0
• Date: September 6, 2024

License

• License: MIT

Developers

• Developed by: Mohammad Shojaei
• Contact: Shojaei.dev@gmail.com

Citation

Model Use

Intended Use

• Primary Use: Tokenization of Persian text for NLP tasks such as text classification, named entity recognition, machine translation, and more.
• Secondary Use: Integration with pre-trained language models like BERT and RoBERTa for fine-tuning on Persian datasets.

Out-of-Scope Use

• Non-Persian Text: This tokenizer is not designed for languages other than Persian.
• Non-NLP Tasks: It is not intended for use in non-NLP tasks such as image processing or audio analysis.

Data

Training Data

• Dataset: mshojaei77/PersianTelegramChannels
• Description: A rich collection of Persian text extracted from various Telegram channels. This dataset provides a diverse range of language patterns and vocabulary, making it suitable for training a general-purpose Persian tokenizer.
• Size: 60,730 samples

Data Preprocessing

• Normalization: Applied NFD Unicode normalization, removed accents, converted text to lowercase, stripped leading and trailing whitespace, and removed ZWNJ characters.
• Pre-tokenization: Used whitespace pre-tokenization.

Performance

Evaluation Metrics

• Tokenization Accuracy: The tokenizer has been tested on various Persian sentences and has shown high accuracy in tokenizing and encoding text.
• Compatibility: Fully compatible with Hugging Face Transformers, ensuring seamless integration with advanced language models.

Known Limitations

• Vocabulary Size: The current vocabulary size is based on the training data. For very specialized domains, additional fine-tuning or training on domain-specific data may be required.
• Out-of-Vocabulary Words: Rare or domain-specific words may be tokenized as unknown tokens ([UNK]).

Training Procedure

Training Steps

1. Environment Setup: Installed necessary libraries (datasets, tokenizers, transformers).
2. Data Preparation: Loaded the mshojaei77/PersianTelegramChannels dataset and created a batch iterator for efficient training.
3. Tokenizer Model: Initialized the tokenizer with a BPE model and applied normalization and pre-tokenization steps.
4. Training: Trained the tokenizer on the Persian text corpus using the BPE algorithm.
5. Post-processing: Set up post-processing to handle special tokens.
6. Saving: Saved the tokenizer to disk for future use.
7. Compatibility: Converted the tokenizer to a PreTrainedTokenizerFast object for compatibility with Hugging Face Transformers.

Hyperparameters

• Special Tokens: [UNK], [CLS], [SEP], [PAD], [MASK]
• Batch Size: 1000 samples per batch
• Normalization Steps: NFD, StripAccents, Lowercase, Strip, Replace (ZWNJ)

How to Use

Installation

Loading the Tokenizer

Tokenization Example

Acknowledgments

• Dataset: mshojaei77/PersianTelegramChannels
• Libraries: Hugging Face datasets, tokenizers, and transformers

References

• Hugging Face Tokenizers Documentation
• Hugging Face Transformers Documentation