sample-10BT version of the FineWeb dataset tokenized using the gpt2 tokenizer and split into 100M tokens binary shards.
A shard is simply a 1D stream of np.uint16 numbers which are the tokenized samples from the dataset, stored contiguously.
Each sample from the dataset was prefixed with the <|endoftext|> special token before being tokenized.
There are 103 training shards (under train/ dir) and 1 shard for validation (under val/).
