This dataset contains fine-grained results from our HPLT 3.0 pre-release evaluations comparing different data deduplication stategies for the pre-HPLT 3.0 corpora with the previous HPLT 2.0 version. We compare the following data deduplication strategies to guide our design choices, and guard against data quality regression compared to HPLT 2.0: pre-HPLT 3.0 CD (per-crawl deduplication), pre-HPLT 3.0… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/2505-deduplication-evals.
