Views
No views yet
identity rule) — indentation preserveddef, class, fn, struct, impl, return, async, …),
operators (==, !=, =>, ->, ::, ///, …) and structural tags
(<python>, <code>, <explanation>, …) are all guaranteed single tokens.1import sentencepiece as spm
2sp = spm.SentencePieceProcessor()
3sp.Load("cofos_tokenizer.model")
4print(sp.EncodeAsPieces("def hello():\n return 42"))