You can deploy this model and start earning money today!

theformatisvalid/
LentaBPE-v15k-f2

Name: LentaBPE-v15k-f2
Availability: InStock

mit

Views

No views yet

[LentaBPE-v15k-f2]

🗃️ Corpus

78k+ words from lenta.ru (2025)

⚙️ Parameters

Algorithm: BPE
Vocabulary size: 15,000
Min frequency: 2

📊 Metrics

OOV rate: 22.21%
Reconstruction accuracy: 90%
Compression ratio: 6.231

💻 Use case

python

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("theformatisvalid/LentaBPE-v15k-f2")
tokens = tokenizer.tokenize("Привет, как дела?")