Papers
arxiv:2611.20003

The novelty exposed: ablation, reproducibility and transparency in 40 technical reports of open-weight large language models (LLMs)

Published on Nov 15
· Submitted by
bahaeddin Türkoğlu
on Nov 18

Abstract

In 2025 alone, 48 new large language model (LLM) architectures were added to the Hugging Face transformers library, and many of them arrive with a technical report that presents the model as new. What is actually new in these models, and is it tested? We put this question to LLM technical reports. We look at three things: what the reference code shows about how new open-weight architectures differ from earlier ones; what 40 randomly sampled technical reports claim as new and whether they support it with ablation studies; and what they disclose for reproducibility and transparency. In the library, 39 of the 48 architectures added in 2025 take at least two of their three core blocks from another model, most often Llama, although part of this reflects how the library writes code. In the reports, 19 of 73 architectural or training novelty claims come with a controlled with/without comparison; 4 of 40 reports state more than one training run; 8 rerun the models they compare against under one setup; and 1 discloses data mixture, token count, and compute together. We also looked for known components given new names: none of five candidate cases hid its origin, though we did not study model names. We propose seven reporting rules for LLM release reports and apply them to two reports. We do not judge model quality, and an untested claim may still be true.

Community

Paper author Paper submitter

We coded 40 randomly sampled LLM technical reports for what they claim as new and how they test it. Only 19 of 73 architectural or training novelty claims come with a controlled with/without ablation, 4 of 40 reports state more than one training run, and 1 discloses data mixture, token count and compute together. In the transformers library, 39 of the 48 architectures added in 2025 reuse at least two of three core blocks from another model. We propose seven reporting rules for release reports. All coded claims (with page-anchored quotes) are in the dataset below.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.20003
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.20003 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.20003 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.