sbordt/forgetting-contamination-124M-7B

124M_7B: 124M GPT-2 model trained on 7B tokens with benchmark contamination

Checkpoint from the ICML 2025 paper How much can we forget about Data Contamination? by Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko and Ulrike von Luxburg. Code and further details: https://github.com/tml-tuebingen/forgetting-contamination.

Run	124M_7B
Architecture	GPT-2, d12 in llm.c: 12 layers, 12 heads, width 768
Training data	FineWeb-Edu sample of 7B tokens with benchmark questions inserted (see the paper and llm.c/create_contaminated_dataset.py)
Training steps	13351
Optimizer	AdamW, betas (0.9, 0.95), weight decay 0.1, peak learning rate 6e-4 with linear warmup and decay to 10%
Training script	llm.c/scripts/7B_tokens/124M.sh
Used in	Figure 1(a) of the paper
All checkpoints and benchmark questions	Hugging Face collection

Files

• gpt=d12_step=13351.pth: the llm.c training checkpoint, a dict saved with torch.save with the keys model ("d12"), model_state_dict, optimizer_state_dict, train_loader_state_dict, batch_size (the per-GPU micro-batch) and last_step.
• evals/all-contamination-splits_eval_step=13351.parquet: the output of llm.c/eval_gpt.py for this checkpoint on the 44,000 de-duplicated benchmark questions: one row per question with the columns options, ce_loss, lcp, prediction, label. The file does not store the benchmark name or the split-id of a question; compute_results.ipynb attaches them by aligning the rows with the benchmark datasets (both sorted by length).

Loading the model

GPT and GPTConfig are defined in llm.c/train_gpt.py of the code repository (importing that file also requires wandb and numpy). The following was run with torch 2.2.2 and loads all weights with strict=True; weights_only=True works as well.

import torch
from huggingface_hub import hf_hub_download
from train_gpt import GPT, GPTConfig   # llm.c/train_gpt.py from the code repository

path = hf_hub_download("sbordt/forgetting-contamination-124M-7B", "gpt=d12_step=13351.pth")
checkpoint = torch.load(path, map_location="cpu", weights_only=False)
state_dict = {k.replace("module._orig_mod.", ""): v for k, v in checkpoint["model_state_dict"].items()}

model = GPT(GPTConfig(block_size=1024, vocab_size=50257, n_layer=12, n_head=12, n_embd=768))
model.load_state_dict(state_dict)

This is the same procedure as in llm.c/eval_gpt.py, which evaluates a checkpoint on the contamination splits of the benchmarks (--checkpoint <file> --model d12).

Citation

@inproceedings{bordt2025forgetting,
  author    = {Sebastian Bordt and Suraj Srinivas and Valentyn Boreiko and Ulrike von Luxburg},
  title     = {How much can we forget about Data Contamination?},
  booktitle = {ICML},
  year      = {2025}
}