SPES-7B is a 7B-parameter Mixture-of-Experts (MoE) Large Language Model pretrained using SPES (SParse Expert Sync), a memory-efficient decentralized training framework.
SPES (SParse Expert Sync) is designed for pretraining MoE LLMs across geographically distributed GPU nodes. It addresses memory and bandwidth constraints by training only a subset of experts per node, significantly lowering the individual memory footprint and eliminating the need for full-parameter transmission. SPES-7B achieves competitive performance with centrally trained models under similar computational budgets.
1@article{zhang2026pretraining,
2 title={Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm},
3 author={Zhang, Jinrui and Xiao, Chaodong and Wu, Aoqi and Zhang, Xindong and Zhang, Lei},
4 journal={arXiv preprint arXiv:2602.11543},
5 year={2026}
6}
