Views
No views yet
three-quarter branch of the D24 v6.2
ratio-20 experiment. This repository contains both the terminal midtraining
state and its terminal SFT descendant.midtrain-hf/: final midtraining Transformers model (BF16 base model).megatron-midtrain/iter_0003200/: exact terminal Megatron
checkpoint, including optimizer/training state.megatron-midtrain/: trackers, completion receipt, and frozen config.provenance/: frozen campaign tickets and SFT receipt/config.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3repo = "sfanm/d24-v6.2-0.75"
4sft_model = AutoModelForCausalLM.from_pretrained(repo, token=True)
5sft_tokenizer = AutoTokenizer.from_pretrained(repo, token=True)
6
7midtrain_model = AutoModelForCausalLM.from_pretrained(
8 repo, subfolder="midtrain-hf", token=True
9)| Stage | Source/final state |
|---|---|
| Pretraining | iteration 64,000; 268,435,456,000 tokens |
| Midtraining | iteration 3,200; 13,421,772,800 tokens |
| SFT | iteration 1,773; 475,114,114 packed tokens |
warmup_iters=134 and
decay_iters=640 with
cosine decay. SFT used a constant learning rate
of 1e-4 from optimizer step zero with zero warmup.