A 60GB, knowledge-dense pretraining corpus built as a clean, superior alternative to TinyStories.
TinyStories teaches grammar. This teaches knowledge, reasoning, and code.
File: pretrain.csv - 60GB, ~15M rows, single column text
TinyStories is fiction for kids. This is textbook / encyclopedia / reasoning / code. Every row is 3500-5200 chars packed with 2-3 fused concepts, not a story.
Physics, Chem, Bio, Math, CS, History, Geo… See the full description on the dataset page:
https://huggingface.co/datasets/Gugu8/Pretrain.
