FineWeb-Mask is a 1.5 trillion token, high-efficiency pre-training dataset curated using the DATAMASK framework. Developed by the ByteDance Seed team, DATAMASK addresses the fundamental tension in large-scale data selection: the trade-off between high quality and high diversity.
By modeling data selection as a Mask Learning problem, we provide a derivative of the original… See the full description on the dataset page:
https://huggingface.co/datasets/DATA-MASK/FineWeb-Mask.
