File size: 2,818 Bytes
d2ff3f8 371e22c d2ff3f8 371e22c d2ff3f8 371e22c 341f3fb 371e22c 807cb72 cad814e 371e22c 807cb72 371e22c 341f3fb 371e22c 16e089c 371e22c 16e089c 371e22c 614ad44 371e22c 614ad44 371e22c 614ad44 371e22c 614ad44 371e22c 614ad44 341f3fb 807cb72 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | ---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- arloop
- recurrent
- custom_code
- pretrained
---
# PreNormLoop-0.6B
📚 [Paper](https://arxiv.org/abs/2611.05234) · 💻 [Code](https://github.com/OSU-NLP-Group/ARLoop)
PreNormLoop is a fully-open pretrained looped transformer with PreNorm residual connections, serving as the baseline for ARLoop. This release
contains the final checkpoint of the 0.6B, 180B-token training run.
It is a **base pretrained model**, intended for text completion, which can be used for mid-training and post-training.
## Load and generate
```bash
pip install 'transformers==5.22.0' safetensors
# Install a PyTorch build suitable for your CPU or CUDA environment separately.
```
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "osunlp/PreNormLoop-0.6B"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True, dtype=torch.bfloat16,
).to(device).eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to(device)
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
Weights load in BF16 for inference. The stored checkpoint retains the original FP32 weights; normalization statistics use FP32 internally.
## Architecture
| Property | Value |
| --- | --- |
| Physical transformer layers | 18, shared across 3 recurrent passes |
| Effective transformer depth | 54 |
| Hidden / MLP intermediate width | 1536 / 4608 |
| Attention / KV heads | 16 / 4 |
| Context length | 4096 |
| RoPE base | 50000 |
| Residual connections | Standard PreNorm additions; no attention-residual routing |
| Value embeddings | Shared across alternating physical layers |
| Tokenizer vocabulary | 49152, SmolLM2 tokenizer used in training |
| Output head | Tied to token embeddings |
## License
Model release: [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
## Training checkpoints
`main` contains the final checkpoint.
| Optimizer step | Training tokens | Revision |
| ---: | ---: | --- |
| 171,660 (final) | 179,998,556,160 | [main](https://huggingface.co/osunlp/PreNormLoop-0.6B/tree/main) |
## Citation
```bibtex
@misc{yao2026arloop,
title={{ARLoop}: Fully-Open Pre-trained Looped Transformers with Attention Residuals},
author={Yuekun Yao and Yupei Du and Dawei Zhu and Huan Sun},
year={2026},
eprint={2611.05234},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2611.05234}
}
```
|