File size: 1,698 Bytes
86c1403 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | ---
pipeline_tag: text-generation
---
# Parcae-Large-770M
Parcae is a novel stable, looped architecture that addresses instability issues in prior looped language models by recasting looping as a nonlinear time-variant dynamical system over the residual stream. This specific checkpoint is the 770M parameter version, trained on the FineWeb-Edu dataset.
- **Paper:** [Parcae: Scaling Laws For Stable Looped Language Models](https://arxiv.org/abs/2604.12946)
- **Repository:** [https://github.com/sandyresearch/parcae](https://github.com/sandyresearch/parcae)
- **Project Page:** [https://sandyresearch.github.io/parcae/](https://sandyresearch.github.io/parcae/)
## Installation
To use this model, you can install the official package:
```bash
pip install parcae-lm
```
## Usage
You can load the pretrained weights and instantiate the model using the following code:
```python
import parcae_lm
# Load the pretrained model from HuggingFace
model = parcae_lm.from_pretrained("SandyResearch/parcae-large-770m")
```
## Model Details
This model variant has the following configuration:
| Model | Parameters | Prelude Layers | Core Layers | Coda Layers | Model Dim | Default Recurrence |
|-------|-----------|---------|------|------|-----------|------------|
| Parcae-770M | 770M | 6 | 6 | 6 | 1280 | 8 |
## Citation
```bibtex
@misc{prairie2026parcaescalinglawsstable,
title={Parcae: Scaling Laws For Stable Looped Language Models},
author={Hayden Prairie and Zachary Novack and Taylor Berg-Kirkpatrick and Daniel Y. Fu},
year={2026},
eprint={2604.12946},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2604.12946},
}
``` |