The sparrow project aims to help beginner to understand the base architecture of a large language model from scratch. Not only the model, but also the optimization methods that are widely use to shorten the training process.
tokenizer from scratch & merge tokenizer
model modules from scratch & train the stacked model
supervised fine-tuning
Reward… See the full description on the dataset page:
https://huggingface.co/datasets/TerenceLau/sparrow.
