Fixed task set
The same Selected_1000 task IDs and dataset revision are used for every group. Each score is successes ÷ 1,000; failed or unknown tasks stay in the denominator.
RESEARCH LEADERBOARD · SEPTEMBER 2026
Same models.
Two ways to solve.
Compare single-pass agents with a DeepAgents harness on real econometric replication tasks.
—models
—experiment groups
—tasks per group
Published 17 Sep 2026
Single-pass Agent
One code-generation call, followed by execution. No iterative agent loop.
Agent + Harness
DeepAgents plans, uses tools and revises code. Up to 6 model calls, plus final execution.
01 / RESULTS
Agent & Harness leaderboard
Locally scored, archived results. Official scorer parity is not yet verified. Different budgets and run protocols mean the lift is descriptive, not an equal-cost comparison.
Loading archived results…
Results could not be loaded. Please retry or open the results archive.
No matching models.
Every score uses all 1,000 tasks. Unknown and invalid outcomes earn no successes.
02 / HOW TO READ THIS
A comparison you can inspect.
Two scoring profiles
Full replication is the local paper metric. The four HF-style metrics reproduce the published descriptions locally; they are not yet verified against the official scorer.
Read lift with the budgets
Baseline: 1 model call. DeepAgents: up to 6 model calls and 4 trial tool calls. Recovery amendments, time limits and run protocols differ; these are not equal-cost causal estimates.
Evidence status & metric definitions
“Metric unknown” means the selected metric cannot be assessed, including failed execution or unavailable outputs. It is different from a task whose final status is unknown. Neither is dropped or counted as a success.
| Model | Single-pass Agent | Agent + Harness |
|---|
HF-style counts are checked against archived task flags. Full-replication counts for Sol and Opus come from accepted summaries; the other four models also have per-task local flags. This refresh does not rerun or rescore any experiment.
Per-run archive files remain frozen, including their original publication metadata. This page is a new display derived from those archives. GPT-5.5 and unarchived experiments are not included in this edition.