You can deploy this model and start earning money today!

amirhofo/
Persian-BPE-Tokenizer

Name: Persian-BPE-Tokenizer
Availability: InStock

Views

No views yet

Persian BPE Tokenizer (30K)

A Byte-Pair Encoding (BPE) tokenizer with a vocabulary size of 30,000, trained on ~2M Persian texts with an average length of 10,000 characters for NLP tasks.

Usage

Encoding

python

from tokenizers import Tokenizer
tokenizer= Tokenizer.from_file("Persian_BPE_Tokenizer_30K.json")
encoded_text= tokenizer.encode("این یک متن آزمایشی است.")
print("Tokens:", encoded_text.tokens)
print("IDs:", encoded_text.ids)

Decoding

python

decoded_text= tokenizer.decode_batch([[id] for id in encoded_text.ids])
print("Decoded:", decoded_text)

Training Data

This tokenizer was trained on the following datasets:

Wikipedia (20231101.fa): https://huggingface.co/datasets/wikimedia/wikipedia
Persian Blog: https://huggingface.co/datasets/RohanAiLab/persian_blog
HomoRich: https://huggingface.co/datasets/MahtaFetrat/HomoRich-G2P-Persian

License

Code and tokenizer: MIT License

Evaluation Metrics

UNK Rate: 0.0% (on 100,000 samples)
Compression Ratio: 4.56 (on 100,000 samples)

Requirements

For using the tokenizer:
- Python >= 3.9
- tokenizers
For training the tokenizer:
- pandas
- datasets
- requests
- hazm

amirhofo/Persian-BPE-Tokenizer