BPE-32k SlimPajama-3M

Tokeniser details

• Back-end: SentencePiece's SentencePieceTrainer.
• Front-end: TkTkT's BPEVocabulariser

• During training: TkTkT's SentencePiecePreprocessor
• During inference: TkTkT's ModernEnglishPreprocessor
  1. NFKC normalisation
  2. Punctuation splitter, whitespace splitter, English contraction splitter
  3. GPT-2's pseudo-byte mapping
  4. Start-of-word marker Ġ
  5. Digit and hyphen isolation

Training details

• Preprocessing and counting the 3M corpus: 2h45m
• BPE merges: 25m

• Examples considered: 3 000 000
• Examples used: 2 609 893 (390 107 examples dropped for being > 8192 characters).
• Characters counted: 6 685 212 190
• Unique words after whitespace splitting: 9 254 839