Views
No views yet
| Metric | Value |
|---|---|
| Algorithm | SentencePiece BPE |
| Vocabulary Size | 32,000 subwords |
| Vocabulary Utilization | 91.09% |
Unknown Token Rate (<unk>%) | 0.0000% |
| Avg Tokens / Sequence | 160.3 |
| Compression Ratio | 3.653 chars/token |
| Token Entropy | 11.64 bits |
| Token | ID |
|---|---|
<pad> | 0 |
<unk> | 1 |
<s> | 2 |
</s> | 3 |
<meitei> | 4 |
<bengali> | 5 |
<romanized> | 6 |
<mask> | 7 |
1import sentencepiece as spm
2
3sp = spm.SentencePieceProcessor()
4sp.Load("tokenizer.model")
5
6text = "ꯃꯅꯤꯄꯨꯔꯤ ꯂꯣꯟ ꯑꯁꯤ ꯑꯆꯧꯕ ꯂꯣꯟ ꯑꯃꯅꯤ"
7tokens = sp.Encode(text, out_type=str)
8print(tokens)
9
10# Decode back
11decoded = sp.Decode(tokens)
12print(decoded)| File | Description |
|---|---|
tokenizer.model | SentencePiece binary model (BPE) |
tokenizer.vocab | Full vocabulary with log-probabilities |
tokenizer_config.json | Training metadata, special tokens, and evaluation metrics |
special_tokens_map.json | HF-compatible special token mapping |
tokenizer_qualitative_samples.json | 50 qualitative encoding/decoding samples for manual inspection |
<unk> guarantee)<meitei>, <bengali>, <romanized> for script-conditioned generation1@misc{manipurigpt_tokenizer_v10,
2 author = {ManipuriGPT Team},
3 title = {ManipuriGPT-Tokenizer-v1.0: SentencePiece BPE Tokenizer for Manipuri},
4 year = {2026},
5 publisher = {Hugging Face},
6 url = {https://huggingface.co/nanskong/ManipuriGPT-Tokenizer-v1}
7}