gpjt/

fineweb-gpt2-tokens

This is all of the 10B sample of FineWeb, split into 99% training data, 1% validation data, each converted to a single uint16 tensor of GPT-2 tokens. The text for each record in the original dataset was tokenized using tiktoken, then they were all concatenated together, with <|endoftext|> tokens separating them. Code:

Script to download the original dataset Script to do the tokenization

This dataset is derived from the original FineWeb… See the full description on the dataset page: https://huggingface.co/datasets/gpjt/fineweb-gpt2-tokens.