The loadable model at this repository's root is the terminal pretraining checkpoint in the public D24 v6 lineage: a 756,819,456-parameter decoder trained from scratch on the nominal 400B-token ClimbMix corpus. The run consumed exactly 354,530,270,862 training tokens and ended at optimizer iteration 84,527.

The Transformers class is LlamaForCausalLM, but this is a from-scratch D24 model with the GPT-2 tokenizer, not a Llama-family pretrained checkpoint.

Training used sequence length 2,048, global batch 2,048, microbatch 4, AdamW at peak LR 4e-4, weight decay 0.1, gradient clip 1.0, and a WSD schedule with 845 warmup steps and a final 16,905-step cosine decay to 4e-5. Final held-out language-model loss was 2.256648 (perplexity 9.551).

This is a base language model and has no chat template. All 23 retained, resumable Megatron distributed checkpoints are published under megatron/, from iter_0004000 through the terminal iter_0084527 (including the deadline checkpoint iter_0067040).

This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.