nthngdy/cplm-baselines-135M

Vanilla 135M baselines for Copy-Paste Language Models

135M-parameter vanilla language models used as baselines in the paper Copy-Paste Language Models (Nathan Godey & Yoav Artzi, Cornell University). Paper

The Copy-Paste counterparts (CPLMs), trained with the exact same recipe, are available at nthngdy/cplm-135M.

Checkpoints

Each model is stored on its own branch. Three backbones are available:

Backbone	Description
full	Full attention in all 30 layers
hybrid	3 sliding-window (w=256) layers for every full-attention layer
sw	Sliding-window attention (w=256) in all layers

These are the final checkpoints (20B tokens, step 19,000, after the LR cooldown).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nthngdy/cplm-baselines-135M"
tok = AutoTokenizer.from_pretrained(repo, revision="full")
model = AutoModelForCausalLM.from_pretrained(repo, revision="full")

ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=32)[0]))

Training

• Architecture: SmolLM3 with 30 layers, hidden size 576, 9 query / 3 key-value heads, tied embeddings and a 49,152-token vocabulary.
• Data: 20B tokens of FineWeb-Edu, 2,048-token sequences.
• Recipe: AdamW (betas 0.9/0.95, weight decay 0.01), peak LR 3e-3, WSD schedule (500 warmup steps, 1,900 cooldown steps, 19,000 steps in total) and batch size 512.

Citation

@misc{godey2026copypastelanguagemodels,
      title={Copy-Paste Language Models},
      author={Nathan Godey and Yoav Artzi},
      year={2026},
      eprint={2610.19247},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.19247},
}