
architecture	`looped-moe`
d_model	640
effective layers	32
loop shape	8 layers x 4 passes
experts	16 total, 2 active
compute budget	1e18 FLOPs
training hardware	A100-80GB
training steps	45,632
parameters (stored)	289,854,720
peak LR	0.005
batch size	16
muP width_ratio	2.5 (d_base=256)
final val loss	n/a

This width is the architecture's own measured A100 minimum on the 1e18 isoFLOP sweep. Architectures optimise at different widths; at fixed compute a wider model simply trains on fewer tokens (C = 6·N_act·D), so this is the best that architecture does with the budget. Do not compare these against the B200 repos (base-32L-d512-1e18, looped-16x2-d640-1e18, looped-moe-16x2-d512-1e18) — same widths, different hardware.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "ml-ryanlee/looped-moe-8x4-e16k2-32L-d640-1e18-a100", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")

Pass max_length=1024 when evaluating — the RoPE buffer is sized to the training context.