README.md

---
language:
  - en
tags:
  - pytorch
  - causal-lm
  - pythia
  - polypythias
license: apache-2.0
datasets:
  - EleutherAI/pile
  - EleutherAI/pile-preshuffled-seeds
library_name: transformers
arxiv: 2503.09543
---

# PolyPythias

This model is part of the **PolyPythias** suite, an extension of the [Pythia](https://github.com/EleutherAI/pythia) project providing 45 additional training runs across 5 model sizes with 9 different random seeds each. These models enable systematic study of training stability and reproducibility in language models.

## Paper

**[PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs](https://arxiv.org/abs/2503.09543)**

Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, and Stella Biderman. *ICLR 2025*.

## Model Details

| Size | Parameters | Layers | Model Dim | Heads | Original Model |
|------|------------|--------|-----------|-------|----------------|
| 14M  | 14M        | 6      | 128       | 4     | [pythia-14m](https://huggingface.co/EleutherAI/pythia-14m) |
| 31M  | 31M        | 6      | 256       | 8     | [pythia-31m](https://huggingface.co/EleutherAI/pythia-31m) |
| 70M  | 70M        | 6      | 512       | 8     | [pythia-70m](https://huggingface.co/EleutherAI/pythia-70m) |
| 160M | 160M       | 12     | 768       | 12    | [pythia-160m](https://huggingface.co/EleutherAI/pythia-160m) |
| 410M | 410M       | 24     | 1024      | 16    | [pythia-410m](https://huggingface.co/EleutherAI/pythia-410m) |

All models were trained on 300B tokens from [The Pile](https://pile.eleuther.ai/).

## Naming Convention

- **`pythia-{size}m`** - Original Pythia model (seed 1234)
- **`pythia-{size}m-seed{1-9}`** - PolyPythias variants with different random seeds
- **`pythia-160m-data-seed{1-3}`** - 160M models with only data ordering varied (weight init fixed)
- **`pythia-160m-weight-seed{1-3}`** - 160M models with only weight initialization varied (data order fixed)

The decoupled seed variants (data-seed and weight-seed) allow researchers to separately study the effects of data ordering vs. weight initialization.

## Quick Start

```python
from transformers import GPTNeoXForCausalLM, AutoTokenizer

# Load the final checkpoint
model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/pythia-70m-seed3")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-70m-seed3")

# Generate text
inputs = tokenizer("The quick brown fox", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0]))
```

## Available Checkpoints

Each model provides **154 intermediate checkpoints** saved as Git branches:

| Checkpoint | Training Tokens | Description |
|------------|-----------------|-------------|
| `step0` | 0 | Initialization (before training) |
| `step1`, `step2`, `step4`, ..., `step512` | 2M - 1B | 10 log-spaced early checkpoints |
| `step1000`, `step2000`, ..., `step143000` | 2B - 300B | 143 evenly-spaced checkpoints |

To load a specific checkpoint:

```python
model = GPTNeoXForCausalLM.from_pretrained(
    "EleutherAI/pythia-70m-seed3",
    revision="step50000",  # Any checkpoint step
)
```

## Training Data

All models were trained on The Pile using pre-shuffled data orderings. The shuffled index files for each seed are available at:

**[EleutherAI/pile-preshuffled-seeds](https://huggingface.co/datasets/EleutherAI/pile-preshuffled-seeds)**

This dataset contains `.idx` files for seeds 0-9 used with `MMapIndexedDataset` to load the memory-mapped Pile data in the correct order for each seed.

### Reproducing Training Data Order

To reproduce the exact data ordering used for a specific seed:

1. Download the Pile dataset and tokenize it using the Pythia tokenizer
2. Download the corresponding seed folder from `pile-preshuffled-seeds`:
   ```bash
   # Using huggingface_hub
   from huggingface_hub import snapshot_download
   snapshot_download(
       repo_id="EleutherAI/pile-preshuffled-seeds",
       repo_type="dataset",
       allow_patterns="seed3/*",  # Download only seed3
       local_dir="./pile-seeds"
   )
   ```
3. Use the idx files with GPT-NeoX's `MMapIndexedDataset`:
   ```python
   from dataset import MMapIndexedDataset
   dataset = MMapIndexedDataset(path_prefix, skip_warmup=True)
   ```

For complete training reproduction instructions, see the [Pythia GitHub repository](https://github.com/EleutherAI/pythia).

## All PolyPythias Models

The complete collection is available at: [EleutherAI/polypythias](https://huggingface.co/collections/EleutherAI/polypythias)

### 14M Parameter Models
- [pythia-14m-seed1](https://huggingface.co/EleutherAI/pythia-14m-seed1) through [pythia-14m-seed9](https://huggingface.co/EleutherAI/pythia-14m-seed9)

### 31M Parameter Models
- [pythia-31m-seed1](https://huggingface.co/EleutherAI/pythia-31m-seed1) through [pythia-31m-seed9](https://huggingface.co/EleutherAI/pythia-31m-seed9)

### 70M Parameter Models
- [pythia-70m-seed1](https://huggingface.co/EleutherAI/pythia-70m-seed1) through [pythia-70m-seed9](https://huggingface.co/EleutherAI/pythia-70m-seed9)

### 160M Parameter Models
- [pythia-160m-seed1](https://huggingface.co/EleutherAI/pythia-160m-seed1) through [pythia-160m-seed9](https://huggingface.co/EleutherAI/pythia-160m-seed9)
- [pythia-160m-data-seed1](https://huggingface.co/EleutherAI/pythia-160m-data-seed1) through [pythia-160m-data-seed3](https://huggingface.co/EleutherAI/pythia-160m-data-seed3)
- [pythia-160m-weight-seed1](https://huggingface.co/EleutherAI/pythia-160m-weight-seed1) through [pythia-160m-weight-seed3](https://huggingface.co/EleutherAI/pythia-160m-weight-seed3)

### 410M Parameter Models
- [pythia-410m-seed1](https://huggingface.co/EleutherAI/pythia-410m-seed1) through [pythia-410m-seed9](https://huggingface.co/EleutherAI/pythia-410m-seed9)

## Evaluation Results

Evaluation results for all models are available in the [polypythias-evals](https://huggingface.co/datasets/EleutherAI/polypythias-evals) dataset.

## Limitations

These models are released for research purposes only. They are **not** intended for deployment in production systems.

- **Not instruction-tuned**: These are base language models that predict the next token; they will not follow instructions like ChatGPT
- **May generate harmful content**: The Pile contains diverse internet text that includes biased, offensive, and factually incorrect content
- **English only**: Models were trained primarily on English text
- **No safety filtering**: Outputs are not filtered for safety or accuracy

## License

Apache 2.0

## Contact

For questions about these models, please use:
- [EleutherAI Discord](https://discord.gg/eleutherai) - #release-discussion channel
- [GitHub Issues](https://github.com/EleutherAI/pythia/issues)

## Citation

If you use these models, please cite:

```bibtex
@inproceedings{vanderwal2025polypythias,
    title={PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs},
    author={van der Wal, Oskar and Lesci, Pietro and Muller-Eberstein, Max and Saphra, Naomi and Schoelkopf, Hailey and Zuidema, Willem and Biderman, Stella},
    booktitle={International Conference on Learning Representations},
    year={2025},
    url={https://arxiv.org/abs/2503.09543}
}
```