DCLM is a recently released high quality dataset that uses model-based quality filtering to filter a large subset of common-crawl for similarity to OpenHermes and other instruction-tuning datasets. For reference see the DCLM paper.
The original authors of DCLM did not release fully deduplicated version of their dataset, claiming that full deduplication did not improve performance. The released version was partially deduplicated in shards.
Nevertheless, when performing… See the full description on the dataset page:
https://huggingface.co/datasets/Zyphra/dclm-dedup.
