README.md

---
license: mit
library_name: pytorch
tags:
  - depth-recurrent
  - weight-tied
  - looped-transformer
---

# Depth-Recurrent Transformer (S1, step 49209)

Final recursive checkpoint for the paper *Per-Token Fixed-Point Convergence in
Depth-Recurrent Transformers, and Why Reading It Beats Learning It*.

A depth-recurrent (weight-tied, looped) transformer: prelude blocks, a
weight-tied core looped `r` times with the recursion count sampled per
optimization step during pretraining, then coda blocks. Trained on FineWeb-Edu
for 12.9B tokens.

- Parameters: 85.6M (state dict); ~57.3M active per forward at r=1
- Training tokens: 12,900,106,240
- Step: 49209

## Contents

This is an inference checkpoint (`config` + `state_dict` only; optimizer and RNG
state removed). It is a `torch.save` dict and requires the model code from the
[code release](https://github.com/jlognn/depth-recurrent-convergence) to load.

```python
import torch
from src.model import build_model

ck = torch.load("model.pt", map_location="cpu", weights_only=False)
model = build_model(ck["config"])
model.load_state_dict(ck["state_dict"])
model.eval()

# recursive forward takes a loops= kwarg
logits = model(input_ids, loops=8)
```

Or evaluate directly with the release harness:

```bash
python bench/benchmark.py --ckpt model.pt --tokens data/fwe-val.bin --loops 1 2 4 8 16 32
python bench/allocate.py  --ckpt model.pt --tokens data/fwe-val.bin --diagnostic
```

## Citation

```
@misc{logan2026pertoken,
  title  = {Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers,
            and Why Reading It Beats Learning It},
  author = {Logan, Joe},
  year   = {2026},
  eprint = {arXiv:XXXX.XXXXX}
}
```