This is a new kind of model optimization. It is based on a new method for the analysis of the functional role of layers within the transformer stack, and on layer duplication (self-merging) to increase intelligence.

Open LLM Leaderboard

RYS-XLarge at #1 on the HuggingFace Open LLM Leaderboard.

The method is described in full here: LLM Neuroanatomy: How I Topped the LLM Leaderboard Without Changing a Single Weight. If you use this model or the technique in your work, please cite that write-up (BibTeX at the bottom of this card) rather than only the model card.

How it works, briefly

A Transformer stack is not uniform: early layers read the input into an abstract internal representation, late layers write it back out, and the middle layers do the reasoning. Sweeping every contiguous block [i, j) of a pretrained model and re-running it a second time produces a heatmap of which blocks help and which hurt. The blocks that help are clustered in the middle of the stack, and duplicating one of them makes the model measurably better at reasoning benchmarks.

Combined heatmap

The sweep that produced this model. Each pixel is a full evaluation of one contiguous block (i, j) of Qwen2-72B, re-run a second time: red is improvement, blue is degradation. The green circle marks the configuration that was shipped.

Split by probe, the two tasks disagree about what the stack is doing, which is the interesting part: maths tolerates duplication almost anywhere before layer 60, while EQ shows sharp structure, with good regions walled off by bad ones.

Math and EQ deltas

For RYS-XLarge, the chosen configuration was (45, 52): layers 0-51 run, then layers 45-79 run again, so layers 45-51 execute twice. Seven extra layers near the middle of an 80-layer stack, taking the model from 72B to 78B. There was no fine-tuning, no gradient descent and no weight modification of any kind: the extra layers are byte-identical copies of layers the model already had.

The layout is not universal. Every model family puts its duplicable circuits in a different place, so the sweep has to be re-run per architecture - here is Llama-3-70B-Instruct for comparison:

Llama-3-70B-Instruct

Follow-up posts cover the method on modern architectures and why it works:

No Weights were modified in this process!

Model improvement with layer duplication:

Average IFEval BBH MATH Lvl 5 GPQA MUSR MMLU-PRO
RYS Improvement 2.61% -2.05% 2.51% 8.16% 2.58% 17.72% 0.31%

This model is based on MaziyarPanahi/calme-2.1-qwen2-72b, which in turn was tuned from Qwen2-72B. As this method is orthogonal to fine-tuning, the further finetune from MaziyarPanahi now has the top position: https://huggingface.co/MaziyarPanahi/calme-2.4-rys-78b

The method is written up in full in the blog post linked above. At the time of release, all four top models on the leaderboard were based on the RYS method. Special thanks to my wife, for putting up with me coding in the basement for too many evenings and weekends for months!

This research was supported with hardware from the appliedAI Institute, whose goal is to generate and communicate high-quality knowledge about trustworthy AI.

Quickstart

Here is a code snippet with apply_chat_template to show you how to load the tokenizer and model and how to generate content.

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
    "dnhkng/RYS-XLarge",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("dnhkng/RYS-XLarge")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 44.75
IFEval (0-Shot) 79.96
BBH (3-Shot) 58.77
MATH Lvl 5 (4-Shot) 38.97
GPQA (0-shot) 17.90
MuSR (0-shot) 23.72
MMLU-PRO (5-shot) 49.20

Citing this work

If this model or the layer-duplication technique is useful in your research, please cite the write-up. It is archived on Zenodo with a DOI, so there is an archival reference if a reviewer objects to citing a blog:

@misc{ng2026rys,
  title     = {LLM Neuroanatomy: How I Topped the LLM Leaderboard Without Changing a Single Weight},
  author    = {Ng, David Noel},
  year      = {2026},
  month     = {March},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.24230106},
  url       = {https://dnhkng.github.io/posts/rys/}
}

DOI: 10.5281/zenodo.24230106 (all versions) Β· ORCID: 0009-0009-3006-4068


ADVERTISING BREAK

I’m on the hunt for new challenges and a chance to dive into some exciting research opportunities. Oh, and did I mention I just snagged a top spot on the Open LLM leaderboard? πŸŽ‰

Profile

Innovation enthusiast, AI strategist, and interdisciplinary-tech nerd – that's me! With over a decade of experience in research and project management, my professional journey has been largely shaped by my passion for artificial intelligence and its potential to transform various industries. With a solid background in artificial intelligence and machine learning, coupled with a knack for innovation and problem-solving (and a healthy dose of curiosity), I'm excited to bring my skills to a new team.

Originally from Australia, where I earned my degrees in Organic Chemistry and Biochemistry, I moved to Germany in 2004. My academic pursuit continued with a Ph.D. in Chemistry at the Max Planck Institute of Biochemistry. Today, I leverage my robust educational background and diverse industry experience to drive AI innovations in a wide range of applications. Hobbies? Lots: I've also built the world's most powerful espresso machine and am working to bring GLaDOS to life.


I'm based out of Munich, Germany, but I would be interested in working remotely for a team with more compute than my 2x 4090s 4x H100s (thanks appliedAI!) πŸš€

Reach out via LinkedIn - Dr David Noel Ng

Downloads last month
168
Safetensors
Model size
78B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for dnhkng/RYS-XLarge

Finetunes
3 models
Quantizations
5 models

Spaces using dnhkng/RYS-XLarge 4

Evaluation results