⚠️ Experimental: MoE inside a recurrent loop has never been validated in published research. This is an exploration to test whether sparse expert routing helps or hurts when weights are shared across loop iterations.

Same as ultron-small-baseline, but with MoE replacing the dense FFN in the recurrent block.

The hypothesis: different loop iterations may activate different experts, effectively creating depth-conditional routing. If expert specialization emerges per-depth, MoE could unlock more expressive recurrent computation without proportionally increasing FLOPs.

Same recipe as the baseline — only difference is use_moe=True.

trojan0x/ultron — Full source code, training scripts, and notebook