Views
No views yet
| Token | ID | Usage |
|---|---|---|
<pad> | 0 | Padding |
<s> | 1 | BOS |
</s> | 2 | EOS |
<unk> | 3 | Unknown |
| `< | user | >` |
| `< | assistant | >` |
| `< | system | >` |
| `< | thinking | >` |
| `< | /thinking | >` |
1import sentencepiece as spm
2
3sp = spm.SentencePieceProcessor()
4sp.Load("tokenizer/hpt_tokenizer.model")
5
6# Encode
7ids = sp.EncodeAsIds("Oi, tudo bem?")
8# Decode
9text = sp.DecodeIds(ids)