
architecture	`base`
d_model	512
effective layers	32
compute budget	1e18 FLOPs
training steps	65,960
parameters (stored)	154,223,616
peak LR	0.005
batch size	16
muP width_ratio	2.0 (d_base=256)
final val loss	n/a

This width is the architecture's measured minimum on a ten-rung isoFLOP sweep (d256-d1408) at 1e18 FLOPs.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "ml-ryanlee/base-32L-d512-1e18", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")

Pass max_length=1024 when evaluating — the RoPE buffer is sized to the training context.