A tokenizer maps words from natural language to numbers like 0, 1, 36 through a “vocabulary”. You can think of each number as the page index of a word in a “dictionary”.
You may choose to build your own vocabulary and train a tokenizer. The code can be found in ./scripts/train_tokenizer.py (for learning reference only; unless necessary, there is no need to retrain one yourself, as MiniMind already comes with a tokenizer).… See the full description on the dataset page: https://huggingface.co/datasets/harithoppil/minimind_dataset.
