This is one the decoder-only models trained on the cleaned variant of HPLTv2.0 data

All of these models have 2.15B parameters and follow the llama architecture as seen in the fineweb ablation studies.

We are releasing intermediate checkpoints for each model at intervals of every 1000 training steps in separate branches. The naming convention is checkpoint_00xxxx00: for example, checkpoint_0005000. The checkpoints range from checkpoint_0001000 to checkpoint_0047684 and the latter is in the main branch.