Seed-variance run for Sparse Layers are Critical to Scaling Looped Language Models (arXiv:2605.09165), trained to measure run-to-run noise in the 1e18-FLOP benchmark numbers.


architecture	`moe`
d_model	832
effective layers	16
compute budget	1e18 FLOPs
training steps	46,795
parameters (stored)	485,863,040
peak LR	0.01
batch size	16
data-order seed	44
init seed	42
muP width_ratio	6.5 (d_base=128)

Important: what varies across these four seeds

Only the training data order. The initialization seed is fixed at 42 for all four runs, as is the validation-batch order. The spread across seeds 42-45 therefore measures data-order variance, which is a lower bound on full run-to-run variance — a study that also varied initialization would be expected to show equal or greater spread. Do not read these error bars as total training noise.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "ml-ryanlee/seedvar-moe-1e18-d832-seed44", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")

Evaluated with OLMES 5-shot core_9mcqa::olmes. When evaluating, pass max_length=1024 — the RoPE buffer is sized to the 1024-token training context and longer sequences overflow it.