LoopLM-135M-naive

A 135M parameter dense looped transformer trained from scratch on FineWeb. Built as part of an exploration of looped LLM architectures inspired by the Parcae paper (arXiv:2604.12946).

Trained on Modal H100s as a hobby research project. This is the naive looped variant — a clean baseline without Parcae's LTI stability mechanisms, which were found to underperform at this scale.

Architecture

• Type: Dense looped transformer (recurrent reuse of one block)
• Total params: 135M (134.1M unique trainable)
• d_model: 1024
• Prelude: 4 transformer blocks
• Loop: 1 transformer block, called T ~ Poisson(μ=6) times per sequence
• Coda: 2 transformer blocks
• Attention: GQA (16 query heads / 8 KV heads), head_dim=64, RoPE θ=10000, QK-norm
• FFN: SwiGLU, dense_ffn=2816
• Vocab: 50304 (GPT-2 BPE + padding), tied embeddings

Training

• Dataset: FineWeb (raw, ~4.6B tokens consumed)
• Hardware: 2× H100 on Modal
• Steps: 17,500 (stopped early at checkpoint to fit budget)
• Batch: 262,144 tokens/step
• Optimizer: Muon (matrices) + AdamW (norms, embeddings)
• LR: Muon 0.02, AdamW 3e-4, cosine decay (60% constant + 40% decay to 0.1×)
• Precision: bf16 with fp32 logits

Results

Model	Architecture	Tokens	Val Loss
LoopLM-30M (prior)	Dense (8 layers)	1B	3.91
LoopLM-135M-naive (this)	Dense looped	4.6B	3.95
HobbyLM-130M-MoE (sibling)	MoE (140M total / 62M active)	10B	3.30

Honest Findings

This project explored several Parcae stability mechanisms (A matrix constraints, prelude input normalization, identity-init B matrix, custom optimizer routing). None outperformed naive looping at this scale. The paper's improvements appear to be most relevant at 1B+ parameters and 100B+ tokens, while at 135M / 4.6B tokens, simple recurrent reuse is competitive enough.

Compared to a sibling 130M MoE trained on more data, the dense looped model loses by ~0.65 in val loss — MoE wins on token-efficiency at this scale.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("harims95/LoopLM-135M-naive")
model = AutoModelForCausalLM.from_pretrained(
    "harims95/LoopLM-135M-naive",
    trust_remote_code=True,
    torch_dtype=torch.float32,
)
model.eval()

# Forward pass for next-token logits
inputs = tokenizer("The capital of France is", return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)
next_token_logits = out.logits[0, -1]
next_token = tokenizer.decode(next_token_logits.argmax())
print(next_token)

Note: a HuggingFace-style .generate() is not wired in this release. The model returns logits and you'd write a simple sampling loop manually. Training was focused on the architecture exploration, not deployment.

Reproducibility

Training code: github.com/harims95/LoopLM

The spec.json in this repo captures the exact training config used for this checkpoint.

Acknowledgments

• Built on top of harishsg993010/HobbyLM (training infrastructure, optimizer setup, data pipeline)
• Architecture inspired by Parcae — Scaling Laws For Stable Looped Language Models
• Trained on FineWeb

License

Apache 2.0 — feel free to use, study, or build on. Acknowledgment appreciated but not required.