open_lm/: Contains the OpenLM config and final checkpoint
evals/: Evaluation results for various benchmarks
- core_9mcqa/: Results of 9 multiple choice QA tasks with the OLMES evaluation framework
- mmlu/: MMLU results with the OLMES evaluation framework
- dclm/: Results using the DCLM evaluation framework
- perplexity/: Perplexity results using the huggingface trainer
indices.tar.zst: The indices for the selected documents in each shard of the Corpus-200B dataset used for training. The indices can be extracted with tar --use-compress-program "zstd" -xf indices.tar.zst.

Usage

Selection method: Random sampling
Domain definition: 24 KMeans Clusters
Domain mixture: MMLU

To use this model, you need to install the open_lm library and add from open_lm.hf import * before loading the model with AutoModel.from_pretrained(...).

Citation

bibtex

@article{wettig2025organize,
  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},
  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},
  journal={arXiv preprint arXiv:2502.10341},
  year={2025}
}