README.md

---
license: apache-2.0
tags:
- moe
- mixture-of-experts
- causal-lm
- olmoe
- distributed-training
- decentralized-training
- sparse-sync
language:
- en
pipeline_tag: text-generation
---

# SPES-9B

SPES-9B is a pretrained language model released as part of paper:

**Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm**

## Model Details

- **Model name:** SPES-9B
- **Model type:** Causal language model
- **Parameters:** 9B
- **Framework:** SPES
- **License:** Apache-2.0

## Project Links

- **GitHub:** https://github.com/zjr2000/SPES
- **Paper:** https://huggingface.co/papers/2602.11543

## Intended Use

This model is intended for:

- research on decentralized LLM pretraining
- research on MoE training and synchronization
- experimentation and evaluation of pretrained language models


## Citation

If you use this model, please cite the SPES paper.

```bibtex
@article{zhang2026spes,
  title={Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm},
  author={Zhang, Jinrui and Xiao, Chaodong and Wu, Aoqi and Zhang, Xindong and Zhang, Lei},
  year={2026}
}