Week 2 MoE Seq2Seq (topk routing)
- Best validation loss: 5.5680
- Top-k: 2
- Aux loss coef: 0.01
Artifacts include the trained state dict (model.pt), metrics (metrics.json), per-epoch history (history.csv), and tokenizer files.
Architecture
- Encoder-Decoder Transformer with Sparse MoE layers
- Hash-based routing (deterministic) or Token-choice top-k routing (learned)
- Load balancing auxiliary loss for top-k routing
- Trained from scratch on XSum for abstractive summarization
