| Property | Value |
|---|---|
| Algorithm | BPE (Byte Pair Encoding) |
| Vocabulary size | 16,000 |
| Pre-tokenizer | Metaspace (▁) |
| Normalizer | NFC + Strip |
| Special tokens | <s>, </s>, <unk>, <pad>, <mask> |
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("smc/malayalam-bpe-tokenizer")
4
5text = "മലയാളം ഒരു ദ്രാവിഡ ഭാഷയാണ്"
6tokens = tokenizer.tokenize(text)
7print(tokens)
8
9encoded = tokenizer(text, return_tensors="pt")
10print(encoded)