| pipeline_tag: text-generation | |
| # Parcae: Scaling Laws For Stable Looped Language Models | |
| [**Paper**](https://huggingface.co/papers/2604.12946) | [**Project Page**](https://sandyresearch.github.io/parcae/) | [**GitHub**](https://github.com/sandyresearch/parcae) | |
| Parcae is a novel stable, looped architecture for language models. Unlike traditional fixed-depth architectures that scale by increasing parameter count, looped architectures increase compute (FLOPs) by sending activations through a block of layers in a loop. Parcae addresses training instabilities in prior looped models by recasting looping as a nonlinear time-variant dynamical system and constraining the spectral norm of injection parameters. | |
| This checkpoint is the 140M parameter version of Parcae trained on the FineWeb-Edu dataset. | |
| ## Installation | |
| To use this model, install the `parcae-lm` package: | |
| ```bash | |
| pip install parcae-lm | |
| ``` | |
| ## Usage | |
| You can load the pretrained weights using the following code: | |
| ```python | |
| import parcae_lm | |
| # Load a pretrained model from HuggingFace | |
| model = parcae_lm.from_pretrained("SandyResearch/parcae-small-140m") | |
| ``` | |
| ## Citation | |
| ```bibtex | |
| @misc{prairie2026parcaescalinglawsstable, | |
| title={Parcae: Scaling Laws For Stable Looped Language Models}, | |
| author={Hayden Prairie and Zachary Novack and Taylor Berg-Kirkpatrick and Daniel Y. Fu}, | |
| year={2026}, | |
| eprint={2604.12946}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.LG}, | |
| url={https://arxiv.org/abs/2604.12946}, | |
| } | |
| ``` |