Views
No views yet
PreTrainedTokenizerFast with 5120 tokens trained on a subset of karpathy/fineweb-edu-100b-shuffle, which is itself a subset of HuggingFaceFW/fineweb-edu.1class SpecialTokens:
2 PAD = 0
3 BOS = 1
4 EOS = 2
5 SYSTEM = 3
6 USER = 4
7 ASSISTANT = 5
8
9special_tokens_map = {
10 "<|PAD|>": SpecialTokens.PAD,
11 "<|BOS|>": SpecialTokens.BOS,
12 "<|EOS|>": SpecialTokens.EOS,
13 "<|SYSTEM|>": SpecialTokens.SYSTEM,
14 "<|USER|>": SpecialTokens.USER,
15 "<|ASSISTANT|>": SpecialTokens.ASSISTANT
16}