full branch of the D24 v6.2
ratio-20 experiment. This repository contains both the terminal midtraining
state and its terminal SFT descendant.midtrain-hf/: final midtraining Transformers model (BF16 base model).megatron-midtrain/iter_0004226/: exact terminal Megatron
checkpoint, including optimizer/training state.megatron-midtrain/: trackers, completion receipt, and frozen config.provenance/: frozen campaign tickets and SFT receipt/config.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3repo = "sfanm/d24-v6.2-1"
4sft_model = AutoModelForCausalLM.from_pretrained(repo, token=True)
5sft_tokenizer = AutoTokenizer.from_pretrained(repo, token=True)
6
7midtrain_model = AutoModelForCausalLM.from_pretrained(
8 repo, subfolder="midtrain-hf", token=True
9)| Stage | Source/final state |
|---|---|
| Pretraining | iteration 84,527; 354,530,270,862 tokens |
| Midtraining | iteration 4,226; 17,725,128,704 tokens |
| SFT | iteration 1,773; 475,114,114 packed tokens |
warmup_iters=177 and
decay_iters=845 with
cosine decay. SFT used a constant learning rate
of 1e-4 from optimizer step zero with zero warmup.