Views
No views yet
| Parameter | Value |
|---|---|
| Model Type | Modified Kneser-Ney 5-gram |
| Tokenizer | Tibetan syllable-based (botok-rs SimpleTokenizer) |
| Training Corpus | bo_corpus.txt |
| Pruning | 0 0 1 |
| Tokens | 72,358,144 |
| Vocabulary Size | 83,721 |
| Order | Count | D1 | D2 | D3+ |
|---|---|---|---|---|
| 1 | 83,721 | 0.7208 | 1.0003 | 1.2406 |
| 2 | 1,513,875 | 0.6906 | 1.0405 | 1.3725 |
| 3 | 3,382,032 | 0.7399 | 1.0905 | 1.3764 |
| 4 | 6,201,199 | 0.8066 | 1.1508 | 1.3914 |
| 5 | 7,456,051 | 0.7466 | 1.2988 | 1.5043 |
| Type | MB | Details |
|---|---|---|
| probing | 384 | assuming -p 1.5 |
| probing | 447 | assuming -r models -p 1.5 |
| trie | 180 | without quantization |
| trie | 98 | assuming -q 8 -b 8 quantization |
| trie | 160 | assuming -a 22 array pointer compression |
| trie | 77 | assuming -a 22 -q 8 -b 8 array pointer compression and quantization |
| Metric | Value |
|---|---|
| Peak Virtual Memory | 12,333 MB |
| Peak RSS | 3,641 MB |
| Wall Time | 42.3s |
| User Time | 41.4s |
| System Time | 13.1s |
1import kenlm
2
3model = kenlm.Model("BoKenlm-syl.arpa")
4
5# Score a tokenized sentence
6score = model.score("བོད་ སྐད་ ཀྱི་ ཚིག་ གྲུབ་ འདི་ ཡིན།")
7print(score)BoKenlm-syl.arpa — ARPA format language modelREADME.md — This model card