File size: 1,694 Bytes
25c3a8b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
license: mit
library_name: pytorch
tags:
- depth-recurrent
- weight-tied
- looped-transformer
---
# Depth-Recurrent Transformer (S1, step 49209)
Final recursive checkpoint for the paper *Per-Token Fixed-Point Convergence in
Depth-Recurrent Transformers, and Why Reading It Beats Learning It*.
A depth-recurrent (weight-tied, looped) transformer: prelude blocks, a
weight-tied core looped `r` times with the recursion count sampled per
optimization step during pretraining, then coda blocks. Trained on FineWeb-Edu
for 12.9B tokens.
- Parameters: 85.6M (state dict); ~57.3M active per forward at r=1
- Training tokens: 12,900,106,240
- Step: 49209
## Contents
This is an inference checkpoint (`config` + `state_dict` only; optimizer and RNG
state removed). It is a `torch.save` dict and requires the model code from the
[code release](https://github.com/jlognn/depth-recurrent-convergence) to load.
```python
import torch
from src.model import build_model
ck = torch.load("model.pt", map_location="cpu", weights_only=False)
model = build_model(ck["config"])
model.load_state_dict(ck["state_dict"])
model.eval()
# recursive forward takes a loops= kwarg
logits = model(input_ids, loops=8)
```
Or evaluate directly with the release harness:
```bash
python bench/benchmark.py --ckpt model.pt --tokens data/fwe-val.bin --loops 1 2 4 8 16 32
python bench/allocate.py --ckpt model.pt --tokens data/fwe-val.bin --diagnostic
```
## Citation
```
@misc{logan2026pertoken,
title = {Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers,
and Why Reading It Beats Learning It},
author = {Logan, Joe},
year = {2026},
eprint = {arXiv:XXXX.XXXXX}
}
```
|