Views
No views yet
mlp_in in every block.
Trained as a successor to aemack-org/cayley-10b
with better val loss and the 32k-2L hierarchy.| Parameter | Value |
|---|---|
| n_layer | 12 |
| n_head | 8 |
| n_embd | 1024 |
| block_size | 1024 |
| vocab_size | 50304 |
| bias | False |
| norm | RMSNorm (affine) |
| MLP | GELU, 4x expansion |
| tokenizer | GPT-2 (tiktoken) |
| dtype | bfloat16 |
| sparsity_mode | cayley |
| cayley_locations | mlp_in |
| cayley_levels | 10,16,0; 15,32,256 |
| cayley_per_parent_budget | True |
| cayley_score_standardize | True |
cayley_levels entry [L, k, delta] per row defines a level with m = 2**L
features, selecting top-k per token (with per-parent budget delta at child
levels). For 10,16,0; 15,32,256 that is L0 = (1024 features, k=16) →
L1 = (32768 features, k=32, delta=256).| Parameter | Value |
|---|---|
| optimizer | Muon (hidden 2D) + AdamW (embeddings) |
| muon_lr | 0.001 |
| muon_min_lr | 5e-05 |
| adamw_lr | 0.001 |
| adamw_min_lr | 5e-05 |
| lr_schedule | linear_warmdown (warmdown_frac=0.7) |
| batch_size | 24 |
| seq_len | 1024 |
| grad_accum_steps | 64 |
| max_iters | 6358 |
| tokens seen | ~10.0B |
| dataset | FineWeb-Edu-100B (10B-token slice) |
| best_val_loss | 3.1162 |
aemack-org/cayley-10b. Same backbone, different CayleySAE
hierarchy (2L / 32k leaves vs. the original 3L / 65k), larger training budget,
and score standardization enabled. Target: beat the 3.173 CE val loss of
cayley-10b at an equal-or-larger token budget, to establish a stronger
interpretability subject model.