Sparse-MoE LM with ~200M active / ~616M total parameters, trained from-scratch on Ultra-FineWeb (English split), tokenizer Qwen/Qwen3-0.6B-Base (vocab 151,936).

Step: 365000
Tokens consumed: 11,960,320,000 (11.960B)
Architecture: see model.py and config.json. 1 shared + 16 routed experts top-2, GQA(10/2)×16 layers, partial RoPE, QK-Norm, SwiGLU, tied embeddings.
Trained autonomously by the ml-intern Claude Code skill.

Run summary

key	value
param_count_total	616,471,632
param_count_active	~203,591,000 (`sum(p.numel())` minus inactive routed experts)
dataset	openbmb/Ultra-FineWeb (split=en, content)
tokenizer	Qwen/Qwen3-0.6B-Base (vocab=151,936)
step	365000
tokens_consumed	11,960,320,000

Caveats

Trained on Ultra-FineWeb-en only — does not speak non-English / code well.
Intermediate checkpoint at ~10B tokens (near-Chinchilla for 200M active, 2.3× over for total params). Downstream evals attached in the run repo (EVAL_*.md).
Predecessor 100M-active run plateaued around 10B tokens on this dataset. This 200M-active model is the scaling follow-up.

AlexWortega/moe-200m-qwen3-step365000-12B-20260525-2225

MoE-200M-active / 616M-total @ 11.96B tokens — ~12.0B-token intermediate ckpt

Run summary

Caveats

AlexWortega/
moe-200m-qwen3-step365000-12B-20260525-2225