Views
No views yet
| Parameters | 253,872,128 (~254M) |
| Architecture | decoder-only transformer — RoPE, RMSNorm (pre-norm), SwiGLU MLP |
| Dims | dim 1024 · 16 layers · 16 heads (head_dim 64) · FFN mult 2.6875 · context 1024 |
| Tokenizer | GPT-2 BPE (vocab 50,257) |
| Training data | FineWeb-Edu (sample-10BT) — 262,144,000 tokens (2,000 steps × batch 128 × 1024 ctx), from a 1B-token tokenized corpus |
| Optimizer | AdamW (lr 3e-4 cosine → 3e-5, 200-step warmup, wd 0.1, β 0.9/0.95), grad clip 1.0, bf16 |
| Final validation loss | 3.8163 (bf16) |
| Compute | ~69 minutes on a single H100 |
RalphBase architecture defined in the
recipe repo (config.json ships the exact recipe
config). Clone the recipe repo for the model class, then load model.safetensors into it.recipe-vX.Y.Z king lineage. The first two autonomous king
changes — recipe-v0.1.0 (warmup-cut) and recipe-v0.1.1 (depth-scaled residual init) —
improve on this baseline. See the
recipe releases and the
Ralph research log.