The MiniPile Challenge for Data-Efficient Language Models
Dataset Summary
MiniPile is a 6GB subset of the deduplicated The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters.
The primary motivation for curating MiniPile is that (i) diverse… See the full description on the dataset page: https://huggingface.co/datasets/JeanKaddour/minipile.
