Views
No views yet
kas) trained on
KS-LIT-3M, a 3.1M-word literary corpus. Released as part of the
KashTok study (Malik et al., 2026), the first systematic
linguist-verified tokenization comparison for Kashmiri.1from transformers import AutoTokenizer
2tokenizer = AutoTokenizer.from_pretrained("Omarrran/Kashmiri_WordPiece_Tokenizer")
3
4text = "کٲشِر زَبان چھِیہٕ خٲص زَبان"
5encoding = tokenizer(text, return_tensors="pt")
6print(tokenizer.tokenize(text))
7print(encoding.input_ids)| Property | Value |
|---|---|
| Tokenizer type | WordPiece |
| Vocabulary size | 16,000 |
| Training corpus | KS-LIT-3M (2.47M words, 129,672 train segments) |
| Special tokens | [PAD] [UNK] [CLS] [SEP] [MASK] |
| Max sequence length | 512 |
| Pre-tokenization | NFC + KS_CHAR_MAP normalization |
| Metric | Value |
|---|---|
| Composite Quality Score (CQS) | 0.8534 |
| Fertility (↓ better) | 1.1951 |
| Diacritic Preservation Score (↑) | 0.9997 |
| Morphological Boundary Alignment (↑) | 0.2759 |
| Out-of-Vocabulary rate (↓) | 0.0000 |
| Reconstruction (char-level, ↑) | 1.0000 |
1@article{malik2026kashtok,
2 title = {KashTok: Tokenizing Kashmiri at Scale with Novel
3 Diacritic- and Morphology-Aware Metrics},
4 author = {Malik, Haq Nawaz and Nissar, Nahfid and others},
5 year = {2026}
6}