File size: 1,522 Bytes
236739c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | ---
pipeline_tag: text-generation
---
# Parcae: Scaling Laws For Stable Looped Language Models
[**Paper**](https://huggingface.co/papers/2604.12946) | [**Project Page**](https://sandyresearch.github.io/parcae/) | [**GitHub**](https://github.com/sandyresearch/parcae)
Parcae is a novel stable, looped architecture for language models. Unlike traditional fixed-depth architectures that scale by increasing parameter count, looped architectures increase compute (FLOPs) by sending activations through a block of layers in a loop. Parcae addresses training instabilities in prior looped models by recasting looping as a nonlinear time-variant dynamical system and constraining the spectral norm of injection parameters.
This checkpoint is the 140M parameter version of Parcae trained on the FineWeb-Edu dataset.
## Installation
To use this model, install the `parcae-lm` package:
```bash
pip install parcae-lm
```
## Usage
You can load the pretrained weights using the following code:
```python
import parcae_lm
# Load a pretrained model from HuggingFace
model = parcae_lm.from_pretrained("SandyResearch/parcae-small-140m")
```
## Citation
```bibtex
@misc{prairie2026parcaescalinglawsstable,
title={Parcae: Scaling Laws For Stable Looped Language Models},
author={Hayden Prairie and Zachary Novack and Taylor Berg-Kirkpatrick and Daniel Y. Fu},
year={2026},
eprint={2604.12946},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2604.12946},
}
``` |