Balochi Tokenizer (hf_bpe, vocab=48000)
Project Overview
This is a research-grade subword tokenizer for Balochi, a low-resource
language spoken by an estimated 15 million people with no prior publicly
available tokenizer. It was selected automatically out of 20 candidate
tokenizers (4 algorithms x 5 vocabulary sizes) using a 10-metric weighted
benchmark.
Dataset Description
- Training corpus: 96821 cleaned Balochi sentences
(1822326 words, 150127 unique word types)
- Evaluation corpus: pre-tokenized / CoNLL-U style gold-token corpus
(28620 sentences) used for morphological boundary agreement
Training Methodology
Trained with hf_bpe at vocabulary size 48000 on identically
cleaned/normalized text (Unicode NFC normalization, deduplication, whitespace
and punctuation normalization) shared across all candidate tokenizers for a
fair comparison.
Evaluation Methodology
16 metrics were computed for every one of the 20 candidate tokenizers:
vocabulary coverage, avg tokens/word, avg tokens/sentence, compression ratio,
vocabulary efficiency, segmentation consistency, UNK rate, tokenization
speed, memory usage, training time, inference latency, model size, token
length distribution, subword frequency, character coverage, and
morphological boundary agreement (vs. CoNLL-U gold tokens). Metrics were
min-max normalized and combined via fixed, pre-declared weights (see
evaluation.csv for full per-tokenizer scores).
Metrics (this tokenizer)
| Metric | Value |
|---|
| algorithm | hf_bpe |
| vocab_size | 48000 |
| training_time_sec | 3.0427 |
| model_size_mb | 3.4769 |
| vocab_coverage_pct | 88.8026 |
| avg_tokens_per_word | 1.1703 |
| avg_tokens_per_sentence | 20.9368 |
| compression_ratio_chars_per_token | 4.0725 |
| unk_token_rate_pct | 0.0000 |
| segmentation_consistency_pct | 100.0000 |
| tokenization_speed_sent_per_sec | 21913.1042 |
| inference_latency_ms_per_sentence | 0.0452 |
| rare_word_avg_subwords | 1.9155 |
| character_coverage_pct | 100.0000 |
| morphological_boundary_agreement_pct | 26.9532 |
| vocabulary_efficiency | 1.8501 |
| memory_delta_mb | 5.9023 |
Limitations
- Trained on 96821 sentences; larger corpora may shift optimal
vocabulary size.
- Morphological boundary agreement is approximate (offset-based), not a
full morpheme-alignment evaluation.
- Domain coverage reflects the source corpus and may not generalize to all
registers of Balochi (e.g. heavy code-switching with Urdu/English).
Intended Uses
Subword tokenization for Balochi language modeling, machine translation,
OCR/ASR text post-processing, and other downstream NLP tasks requiring a
Balochi-aware vocabulary.
License
Apache 2.0
Citation
If you use this tokenizer, please cite the associated GitHub repository:
https://github.com/shah-bakhsh/BalTokenizer.git
