This is a BPE tokenizer based on "Salesforce/codegen-350M-mono".
The tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece)
so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not.
We used ArXiv subset of The Pile dataset and proof steps from
lean-step-public datasets to train the tokenizer.
