allenai/

OLMoE-mix-0924

The following data mix was used to train OLMoE-1B-7B, a Mixture-of-Experts LLM with 1B active and 7B total parameters released in September 2024. The base version of OLMoE-1B-7B can be found at this page, the SFT of OLMoE-1B-7B is available here, and a version combining SFT and DPO is available following this link.

	Statistics




	

Subset Tokens Words Bytes Docs

DCLM Baseline 1.0 3.86 T 3.38 T16.7 T 2.95 B

Starcoder 101 B 63.9 B… See the full description on the dataset page: https://huggingface.co/datasets/allenai/OLMoE-mix-0924.