timaeus/

dsir-pile-13m-filtered-no-github-or-dm_mathematics

This is a filtered version of timaeus/dsir-pile-13m.

Excluded: All rows where metadata.pile_set_name contains 'Github' or 'DM_mathematics' Kept: All other rows from the original dataset

Original: ~13M examples
Filtered: 12,782,200 examples (99.9% of original) Uploaded in: 64 batch files

from datasets import load_dataset dataset =… See the full description on the dataset page: https://huggingface.co/datasets/timaeus/dsir-pile-13m-filtered-no-github-or-dm_mathematics.