j23h67/compute-multipliers-checkpoints

Pretraining progress is mostly coming from data

These are the trained models, evaluation data and code behind our post Pretraining progress is mostly coming from data (Dwarkesh Patel and Jerry Han, 2026).

How much of the progress in pretraining from 2019 to 2025 has come from data versus model improvements? During each of those years, a new open model recipe was published which codified that year's publicly known algorithmic tweaks, and a new public data corpus was released. We train combinations of these year-representative model recipes (GPT-2 to OLMo-2) and data corpuses (OpenWebText to Ultra FineWeb) from scratch, across compute budgets up to 1e19 FLOPs. We evaluate the models on end capabilities as measured by OLMES, which aggregates 10 relatively easy benchmarks, mostly multiple choice QA.

Correction to the post. In the post, we counted OLMo-2's compute using the parameter count of its published model ladder, which also includes the output layer. For every other model recipe, we count only the non-embedding parameters.

We now standardize our counting to count OLMo-2 the same way as well. This makes the model improvements larger than we reported: 5.0x rather than 3.7x. The data improvements (12.0x) are unchanged, and data still accounts for most of the compute efficiency gains. Everything on this page uses the corrected count, and python reproduce/findings.py --labelled-budgets recovers the post's original numbers from the same data. Our error bars come from resampling the training seeds rather than from the post's parametric bootstrap, so they are tighter. The post's 88% for the grid is the same fit as the 89% below (0.888), rounded differently.

Results

We find that from 2019 to 2025, 2.4x more compute efficiency gains have come from data improvements rather than model improvements (12.0x for data and 5.0x for models), at the 1e19 FLOPs compute budget. Improving both gives 21.1x.

With error bars (68% intervals), the compute multipliers are 12.0x [10.4, 13.7] for data, 5.0x [4.1, 5.9] for models and 21.1x [16.4, 26.7] for both, and data improvements gave 2.4x [2.3, 2.5] more compute efficiency than model improvements.

Here is a grid which shows how much better a model we train does on OLMES, relative to the 2019 data + model baseline (GPT-2 trained on OpenWebText), at 3.16e18 FLOPs.

We find that the gains from data and model improvements are mostly independent and don't interact (i.e. realizing the gains from some model improvement doesn't require a specific data corpus, or vice versa). 89% of the variance in the OLMES score can be explained by additive effects of the model and data improvements (using a linear model).

From 2019 to 2025, we observe year-over-year compute efficiency gains of 1.31x [1.27, 1.34] on the model side and 1.51x [1.48, 1.55] on the data side. Measured jointly, we observe a 1.66x year-over-year gain [1.59, 1.73].

Here are the compute scaling curves of every model recipe, trained on FineWeb Edu, and of every data corpus, trained with OLMo-2.

python reproduce/findings.py recomputes every number in this section from the files in this repository.

What is in this repository

Every training run behind the results and figures: 267 runs, 259 of them with model weights. The runs fall into four sets, named as in the sets column of checkpoints.csv.

set	what varies	compute budgets	runs
2019-2025 stack	GPT-2 or OLMo-2, trained on OpenWebText or Ultra FineWeb	1e17 to 1e19	65
model recipes	all 7 model recipes, trained on FineWeb Edu	1e17 to 1e19	105
data corpuses	OLMo-2, trained on all 7 data corpuses	1e17 to 1e19	109
grid	every model recipe on every data corpus	3.16e18	79

Some runs belong to more than one set (82 do). Eight runs have no weights: they were learning rate checks for GPT-J, GPT NeoX, LLaMA and LLaMA 3 at 1e18, and their weights were not kept. Their records and scores are still included.

Finding a checkpoint

Each run has its own folder:

checkpoints/<recipe>/<corpus>/<budget>/<size>-seed<seed>/
    model.safetensors    the model weights (float32)
    run.json             the run's record: model shape, learning rate, tokens, loss during training

For example, checkpoints/v2025_olmo2_control/ultra_fineweb_en/1e19/300M-seed6198/ is OLMo-2 trained on Ultra FineWeb at 1e19 FLOPs with seed 6198. Each configuration has only the model size used in the results. A folder ending in -run2 is a second run of the same seed, and the results average the two.

year	model recipe	data corpus
2019	v2019_gpt2 (GPT-2)	openwebtext (OpenWebText)
2020	v2020_gpt3 (GPT-3)	c4_english (C4 English)
2021	v2021_gptj (GPT-J)	pile_original (The Pile)
2022	v2022_neox (GPT NeoX)	olm_cc_2022 (OLM CC 2022 Snapshot)
2023	v2023_llama (LLaMA)	falcon_refinedweb (Falcon RefinedWeb)
2024	v2024_llama3 (LLaMA 3)	fineweb_edu (FineWeb Edu)
2025	v2025_olmo2_control (OLMo-2)	ultra_fineweb_en (Ultra FineWeb)

Sizes s2 to s9 are the model sizes shared by the 2019 to 2024 recipes, and 16M to 530M are OLMo-2's own model sizes. checkpoints.csv lists every run with its folder, settings and scores, so you can also pick checkpoints by filtering it. To download every checkpoint of one configuration:

from huggingface_hub import snapshot_download
snapshot_download('j23h67/compute-multipliers-checkpoints',
                  allow_patterns=['checkpoints/v2025_olmo2_control/ultra_fineweb_en/1e19/*'])

Other files

checkpoints.csv                 one row per run: folder, set, recipe, corpus, budget, model size, parameters,
                                tokens, compute, seed, learning rate, held-out loss, OLMES and its ten task scores
data/curve_points.csv           which runs make up each point in the figures and each grid cell
data/olmes_items.csv            every OLMES question
data/olmes_item_outcomes.npz    whether each model got each question right
reproduce/                      scripts that recompute the results and figures (CPU, seconds)
eval/                           scripts that re-score a model on OLMES
training/                       the model and training code, model recipes and data corpus settings
load_example.py                 loads a model from its folder

Three columns of checkpoints.csv need explaining. n_params is the number of non-embedding parameters (the attention and MLP weights). flops is the compute used, 6 × n_params × tokens. n_params_planned is the parameter count each run's token budget was planned from. It equals n_params for every model recipe except OLMo-2.

Reproducing the findings

These need only Python with numpy and matplotlib, and can be run locally on CPU.

pip install numpy matplotlib
python reproduce/findings.py        # every number in Results, checked against this page
python reproduce/verify_scores.py   # rebuilds all 267 OLMES scores from the per-question results
python reproduce/figures.py         # redraws the four figures

Going further takes a GPU and the original data corpuses:

• eval/README.md explains how to re-score any model on OLMES and check, question by question, that you get the same results as we did.
• training/README.md explains how to rebuild any of the seven data corpuses from its original source and retrain any run. Its demo retrains one of the models in this repository.

We tested both. Rebuilt data corpuses match the ones we trained on byte for byte. A retrained demo model lands within seed noise of the original. Re-scoring two of the models reproduced all 75,364 of their per-question results.

Notes on the run records

Every run.json contains experiment_stage: "nonreportable_pilot" and reportable: false, and every run ID starts with pilot-. These are just the names of the launch system all our training runs went through. Every model here finished training.

field in run.json	meaning
n_nonembed_block_matrix	same as n_params
n_compute, n_nonembed, n_source, n_compute_campaign	same as n_params_planned
n_realized_trainable_excluding_wte	every trained parameter except the token embeddings
flops_6nd_block_matrix	same as flops
flops_6nd	6 × n_params_planned × tokens, the labelled budget
flops_exact	compute including the output layer and the attention scores
extra_eval	(tokens, loss) on the corpus's held-out text during training; the last value is native_nll
extra_eval_bins	which held-out text was used
curve	loss on a fixed sample from the end of the training data
final_loss	training loss at the end of the run
wall_sec	training time in seconds, on one GPU

Loading a model

import sys
from huggingface_hub import snapshot_download

folder = 'checkpoints/v2025_olmo2_control/ultra_fineweb_en/1e19/300M-seed6198'  # OLMo-2 on Ultra FineWeb, 1e19
root = snapshot_download('j23h67/compute-multipliers-checkpoints',
                         allow_patterns=[f'{folder}/*', 'training/*', 'load_example.py'])
sys.path.insert(0, root)
from load_example import load
model, spec = load(f'{root}/{folder}')

This needs torch, safetensors and pyyaml. The models use the GPT-2 tokenizer (tiktoken.get_encoding('gpt2')) and a context of 2,048 tokens.

Licence and citation

The weights, run records, data files and code are released under Apache-2.0. We used each data corpus under its own terms, and no corpus text is included here. The Pile and OLM CC 2022 restrict redistribution, so check their terms before using the models trained on them for anything beyond research.

@misc{patel_han_2026_pretraining_data,
  title  = {Pretraining progress is mostly coming from data},
  author = {Dwarkesh Patel and Jerry Han},
  year   = {2026},
  url    = {https://www.dwarkesh.com/p/pretraining-progress-is-mostly-data}
}