Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
Abstract
A new benchmark evaluates whether LLM agents improve through repeated professional tasks with feedback, revealing limited sustained learning in current frontier models.
Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience. We introduce Continual Learning Exam (CLE), a large-scale benchmark developed by 50+ domain experts to measure experience-driven improvement in realistic professional workflows. CLE comprises 120 human-authored tasks, each organised as a five-level ladder, yielding 600 levels spanning 89 professional roles across 12 occupational domains. Agents must pass each level to unlock the next, with later levels building on experience accumulated during earlier assignments. An evaluation agent assesses each submission against human-authored rubrics. We compare a single-submission setting with one that provides diagnostic feedback after failure and allows one revision. Across 17 harness-model configurations using frontier LLMs, feedback and revision yield a median completion gain of 5.3 percentage points. Yet the best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks, highlighting the difficulty of sustained learning from experience for existing frontier LLMs. As such, the dataset can also be used to post-train LLMs to strengthen their continual learning capabilities.
Community
🚀 Introducing Continual Learning Exam (CLE): Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents!
🧠 Can LLM agents truly learn from experience and continuously improve over time?
Most existing agent benchmarks focus on whether agents can solve individual tasks. But real-world intelligence requires more than task completion: agents should learn from past successes and failures, accumulate experience, and transfer acquired knowledge, strategies, and skills to future challenges.
To bridge this gap, we introduce Continual Learning Exam (CLE), a large-scale benchmark designed to evaluate long-horizon continual learning and experience-driven self-improvement in LLM agents.
🌟 Key Highlights:
🔹 Built with 50+ domain experts, covering 120 real-world professional tasks, 600 progressive levels, and 89 occupations across 12 domains.
🔹 Realistic professional environments, featuring ~24K real-world files, 19 stateful applications, and 16K+ human-authored evaluation criteria.
🔹 Game-like Task Ladders, where agents must complete each level to unlock the next. Later levels require agents to reuse and build upon knowledge, strategies, and skills acquired through earlier experience.
📊 Key Findings:
We evaluated 17 frontier agent configurations. Even the strongest system, Claude Code + Opus 5, achieves only:
• 47.7% level completion rate
• 23.3% full Task Ladder completion rate
These results reveal a substantial gap between solving individual tasks and achieving genuine long-term continual learning.
We hope CLE can serve as a realistic and verifiable benchmark to advance research on Agent Continual Learning, Memory, Skill Learning, and Recursive Self-Improvement (RSI).
💡 Beyond Task Solving: Towards Continual Learning and Experience-Driven Self-Improvement.
We welcome feedback, discussions, and collaborations from the community!
Get this paper in your agent:
hf papers read 2610.33956 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 4
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper