Pretraining model (base, no SFT/alignment) — 1B Transformer++ trained from scratch (next-token prediction on quality-filtered tiered data: math, web, code, synth, reformat + gold set).

Spec	Value
Params	1,031,898,624 (~1.03B)
Hidden	1536 · 32 layers · 12 attn heads · 4 KV (GQA) · head_dim 128
FFN	4608 SwiGLU
Seq len	2048 (train) / 8192 (config)
Vocab	49152 (SmolLM2-135M tokenizer, BPE)
Optimizer	CautiousAdamW, bf16, cosine LR
Best loss	1.7888 @ step 66253 (115K-cycle hot state: stopped at step 68.8K)

This is a raw pretrained base model — no instruction tuning, no chat template. Suitable for continued pretraining, SFT, or DPO. Weight tying: off (untied embeddings).

kenpeter123/small-1b-pretrain

Small-1B Pretraining Model

kenpeter123/
small-1b-pretrain