epfl-dlab/

llaza-200B

This dataset is the full Llaza pretraining-data mixture for zip2zip language-model pretraining. It combines general web text, code, math, and multilingual web text with byte-based top-level mixture ratios.

Domain Source Target byte ratio

General HuggingFaceFW/fineweb-edu, sample-100BT 50%

Code bigcode/the-stack-dedup 20%

Math HuggingFaceTB/finemath, finemath-3plus 10%

Multilingual epfml/FineWeb2-HQ, 20 language subsets 20%… See the full description on the dataset page: https://huggingface.co/datasets/epfl-dlab/llaza-200B.