Update README.md
Browse files
README.md
CHANGED
|
@@ -19,10 +19,6 @@ SPES-2B is a pretrained language model released as part of paper:
|
|
| 19 |
|
| 20 |
**Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm**
|
| 21 |
|
| 22 |
-
## Overview
|
| 23 |
-
|
| 24 |
-
SPES-2B is a research checkpoint for studying decentralized and memory-efficient pretraining of Mixture-of-Experts (MoE) language models. The model is trained under the SPES framework, which enables distributed training across GPU nodes with lightweight expert synchronization.
|
| 25 |
-
|
| 26 |
## Model Details
|
| 27 |
|
| 28 |
- **Model name:** SPES-2B
|
|
@@ -44,9 +40,6 @@ This model is intended for:
|
|
| 44 |
- research on MoE training and synchronization
|
| 45 |
- experimentation and evaluation of pretrained language models
|
| 46 |
|
| 47 |
-
## Notes
|
| 48 |
-
|
| 49 |
-
This is a research model. Performance may vary depending on evaluation setup, prompting format, and downstream tasks.
|
| 50 |
|
| 51 |
## Citation
|
| 52 |
|
|
|
|
| 19 |
|
| 20 |
**Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm**
|
| 21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
## Model Details
|
| 23 |
|
| 24 |
- **Model name:** SPES-2B
|
|
|
|
| 40 |
- research on MoE training and synchronization
|
| 41 |
- experimentation and evaluation of pretrained language models
|
| 42 |
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
## Citation
|
| 45 |
|