ArXiv | Models | Code
DCLM-pro is refined from DCLM using the ProX refining framework.
It contains about >500B high quality tokens, ready for general language model pre-training.
DCLM-pro is based on DCLM, which is made available under an cc-by-4.0 license.
@article{zhou2024programming,
title={Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale},
author={Zhou, Fan and Wang, Zengzhi… See the full description on the dataset page:
https://huggingface.co/datasets/gair-prox/DCLM-pro.
