Compact sentencepiece tokenizer for sample-efficient English language modeling, simply tokenizing natural language.
1from transformers import AutoTokenizer
2
3tokenizer_baby = AutoTokenizer.from_pretrained("nilq/baby-tokenizer")
1from tokenizers import Tokenizer
2
3tokenizer_baby = Tokenizer.from_pretrained("nilq/baby-tokenizer")
This tokeniser is derived from the BabyLM 100M dataset of mixed domain data, consisting of the following sources:
