The model was trained using NVidia-Megatron-LM (branch v0.15) on the LUMI HPC supercomputer. The training utilized 16 AMD MI250x nodes, totaling approximately 1500 GPU hours.

Global batch size: 512 samples (approx. 2M tokens)
Learning rate: 0.002 (WSD, 2000 iteration warmup, 10% iterations decay)

The full SLURM script used to run training is https://github.com/OpenEuroLLM/datamix-models-lumi/blob/main/train-0.4B-ne.sh. DATA_PATH and the WANDB_* settings were adjusted for the run.

Training logs

The model training logs are available on Weights & Biases: https://wandb.ai/openeurollm-project/datamix/runs/ea48d1dr.

Intermediate Checkpoints

We have released intermediate checkpoints to provide access to the model's training progression. These checkpoints are available in separate branches, with a new checkpoint released every 4000 training steps.

The naming convention is iter_0xxxxxxx. For example, the checkpoint for 16000 iterations is named iter_0016000. The available checkpoints range from iter_0004000 up to iter_0047684. The final checkpoint, iter_0047684, is located in the main branch.

openeurollm/datamix-0.7b-hplt-3.0-eng-100bt

Model Details

Training Data

Tokenizer

Training Information

Training logs

Intermediate Checkpoints

openeurollm/
datamix-0.7b-hplt-3.0-eng-100bt