Views
No views yet
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("mjbommar/ogbert-tokenizer-16k")
4tokens = tokenizer.encode("hello world")<|start|>, <|end|>, <|pad|>, <|unk|>, <|cls|>, <|sep|>, <|mask|>)1@misc{bommarito2025opengloss,
2 title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph},
3 author={Michael J. Bommarito II},
4 year={2025},
5 eprint={2511.18622},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}