d24-pretrain-v3-climbmix-50B
from-scratch 50B-token ClimbMix base (v3 — the largest d24 pretrain).
nanochat-style depth-24 decoder — 24 layers × 1536 hidden × 12 heads, SwiGLU / RoPE / RMSNorm, tied embeddings, GPT-2 BPE vocab (50304), 0.757B params, 2048-token context.
Lineage. from-scratch pretrain on 50B tokens of ClimbMix with a WSD schedule (warmup 700, stable, decay over the last ~9500 of 47684 iters), lr 3e-4→3e-5, micro_batch 8 / global_batch 512 / seq 2048, on NCSA DeltaAI (GH200, multi-node Megatron with the distributed optimizer disabled).
Metrics. held-out val lm-loss 2.375 / PPL 10.75.
Use (base LM)
This is a base language model (pre-SFT) — use it for text continuation, not chat. EOS is the GPT-2 <|endoftext|> (50256). For a chat model, use the d24-sft-* checkpoints.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2mid = "sfanm/d24-pretrain-v3-climbmix-50B"
3tok = AutoTokenizer.from_pretrained(mid)
4model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype="bfloat16", device_map="auto")
5inputs = tok("The derivative of x**2 is", return_tensors="pt").to(model.device)
6print(tok.decode(model.generate(**inputs, max_new_tokens=128)[0], skip_special_tokens=True))
Research checkpoint from a from-scratch nanochat-d24 replication (pretrain → midtrain → SFT → RL) on NERSC Perlmutter. Trained on third-party corpora (ClimbMix, FineMath, OpenMath, MetaMath, OpenThoughts, OLMo-3 Dolmino, SmolTalk, …) — see those datasets' licenses; provided as-is for research.
