from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mechramc/kalavai-phase1-410m-fiction-specialist-seed42")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-410m")

This model is one specialist in a KALAVAI cooperative. To reproduce the MoE fusion results from the paper, load multiple domain specialists and combine them with a trained MoE router (see the paper and GitHub for details).

Citation

bibtex

@article{kalavai2026,
  title={KALAVAI: Cooperative Decentralized LLM Training via MoE Fusion},
  author={[Authors]},
  journal={arXiv preprint arXiv:2603.22755},
  year={2026}
}