DPO (Direct Preference Optimization) model — 1B Transformer++ base, post-trained on top of SFT (kenpeter123/small-1b-sft) using preference pairs. Reference model = policy at init (standard DPO from the SFT anchor). 8-bit Adam, bf16, stage-isolated checkpoint.

Spec	Value
Params	1,031,898,624 (~1.03B)
Hidden	1536 · 32 layers · 12 attn heads · 4 KV (GQA) · head_dim 128
FFN	4608 SwiGLU
Seq len	2048 (train) / 8192 (config)
Vocab	49152 (SmolLM2-135M tokenizer, BPE)
DPO	steps 300 · lr 1e-05 · β 0.1 · sigmoid loss
Optimizer	bitsandbytes Adam8bit, bf16

Preference pairs: raw problem prompt (chosen = gold correct solution, rejected = SFT model's wrong/echo generation). Raw training checkpoint included as dpo_best.pt (model_state_dict only, bf16).

kenpeter123/small-1b-dpo

Small-1B DPO Model

kenpeter123/
small-1b-dpo