This repository contains a deterministic, pretokenized Megatron indexed
dataset built from a pinned revision of the globally shuffled FineWeb-Edu
100BT subset.
The split source files are disjoint, but their text is not content-deduplicated
across splits. The artifact is published unchanged with the measured overlap
preserved:
Held-out split
Exact matches with training
MinHash candidate overlap… See the full description on the dataset page:
https://huggingface.co/datasets/AverageMetaheuristicsEnjoyer/fineweb-edu-gpt2-megatron.
