Papers
arxiv:2611.23068

The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements

Published on Nov 17
· Submitted by
Zeng Yuanhao
on Nov 19
#2 Paper of the day
Authors:
,
,
,
,
,
,
,
,
,
,

Abstract

Agent evolution is often bottlenecked by noisy evaluations that cannot reliably identify better candidate artifacts or rank methods, so progress requires making true improvements detectable and accumulable.

Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.

Community

Paper submitter

TL;DR: Much of the current community effort on agent evolution / RSI (across skills, harnesses, memory, etc.) is focusing on the wrong bottleneck. Proposing better artifacts is actually quite easy—the real challenge is reliably identifying whether an artifact truly helps. In fact, many seemingly impressive gains reported in recent work come down to exploiting evaluation shortcuts.

Everything you want to know about the reality of agent evolution is in this paper. For instance, we mapped out the actual score density distributions across mainstream evolution methods, frontier models, and standard benchmarks (spanning 1,030+ evolution runs and 10M+ LLM calls!). Everyone in the field has wanted to see these curves, though computing each one is prohibitively expensive—so we bit the bullet and absorbed the cost to save the community from hitting the same walls.

Surprise: Many of these score distributions aren't Gaussian at all—they are multimodal!

We also packed the paper with tons of hard-earned, practical insights:

  • How can a single sentence hack mainstream benchmarks and reproduce 90% of full evolution gains?
  • Are there actually transferable skills that genuinely boost performance on Terminal-Bench v2.1? (Spoiler: Yes, and we show why standard selectors miss them.)

Plus plenty more actionable know-how inside. If you are building or evaluating self-evolving agents, you'll definitely walk away with something valuable!

🔗 Project Page: https://lineshogan.github.io/resolution-hypothesis/
20261119-110619

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.23068 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.23068 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.23068 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.