Is this your dataset? Claim it with the Hugging Face account that owns it.

alexkstern/
c4-nanochatbpe-10B

Name: c4-nanochatbpe-10B
Creator: alexkstern

Views

No views yet

c4-nanochatbpe-10B

C4 (en) (from allenai/c4), pre-tokenized with the nanochatbpe tokenizer (vocab 65,536) and packaged as flat uint16 token-id .bin files for fast memmap training.

file split tokens

train.bin train 10,000,000,000

val.bin val 168,272,017

train and val are disjoint held-out partitions. Each .bin is a raw little-endian uint16 stream (no header); token count = filesize / 2, and train.meta.json / val.meta.json carry the full metadata. The tokenizer/files… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/c4-nanochatbpe-10B.