This is a Megatron-LM checkpoint (sharded). To use: Updated Git hooks. Git LFS initialized. Then point Megatron's at the cloned dir; tokenizer is at . Architecture flags must match — see the recipe in the upstream training repo.

Training schedule

Iters 0–12,000: GBS=128, LR 3e-4 cosine, warmup 1000 — over 80,000-step schedule
Iters 12,000–16,000: GBS=512, LR cosine warm-restart over 30,000-step schedule

Tokenizer (chat-extended)

See tokenizer/ for the Cl100kChatTokenizer with 16 reserved chat/think/tool tokens (IDs 100277-100292).

Norapom/experimental_gqa_1_5b

experimental_gqa_1_5b — iter 16,000

Architecture

Loading

Training schedule

Tokenizer (chat-extended)

Norapom/
experimental_gqa_1_5b