ARLoop-0.6B

📚 Paper · 💻 Code

ARLoop is a fully-open pretrained looped transformer with attention residuals. This release contains the final checkpoint of the 0.6B, 180B-token training run. It is a base pretrained model, intended for text completion, which can be used for mid-training and post-training.

Load and generate

pip install 'transformers==5.22.0' safetensors
# Install a PyTorch build suitable for your CPU or CUDA environment separately.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "osunlp/ARLoop-0.6B"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16,
).to(device).eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to(device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Weights load in BF16 for inference. The stored checkpoint retains the original FP32 weights; routing and normalization statistics use FP32 internally.

Architecture

Property	Value
Physical transformer layers	18, shared across 3 recurrent passes
Effective transformer depth	54
Hidden / MLP intermediate width	1536 / 4608
Attention / KV heads	16 / 4
Context length	4096
RoPE base	50000
Attention-residual block size	18 sublayers
Recent residual blocks	3, with input embedding retained
Value embeddings	Shared across alternating physical layers
Tokenizer vocabulary	49152, SmolLM2 tokenizer used in training
Output head	Tied to token embeddings

License

Model release: Apache License 2.0.

Training checkpoints

main contains the final checkpoint.

Optimizer step	Training tokens	Revision
171,660 (final)	179,998,556,160	main

Citation

@misc{yao2026arloop,
  title={{ARLoop}: Fully-Open Pre-trained Looped Transformers with Attention Residuals},
  author={Yuekun Yao and Yupei Du and Dawei Zhu and Huan Sun},
  year={2026},
  eprint={2611.05234},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2611.05234}
}