Views
No views yet
half branch of the D24 v6.2
ratio-20 experiment. This repository contains both the terminal midtraining
state and its terminal SFT descendant.midtrain-hf/: final midtraining Transformers model (BF16 base model).megatron-midtrain/iter_0002200/: exact terminal Megatron
checkpoint, including optimizer/training state.megatron-midtrain/: trackers, completion receipt, and frozen config.provenance/: frozen campaign tickets and SFT receipt/config.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3repo = "sfanm/d24-v6.2-0.5"
4sft_model = AutoModelForCausalLM.from_pretrained(repo, token=True)
5sft_tokenizer = AutoTokenizer.from_pretrained(repo, token=True)
6
7midtrain_model = AutoModelForCausalLM.from_pretrained(
8 repo, subfolder="midtrain-hf", token=True
9)| Stage | Source/final state |
|---|---|
| Pretraining | iteration 44,000; 184,549,376,000 tokens |
| Midtraining | iteration 2,200; 9,227,468,800 tokens |
| SFT | iteration 1,773; 475,114,114 packed tokens |
warmup_iters=92 and
decay_iters=440 with
cosine decay. SFT used a constant learning rate
of 1e-4 from optimizer step zero with zero warmup.