Reproducing SWE-Bench Pro: Can AI Agents Solve Long-Horizon SE Tasks?
The paper reports 1,865 problems across 41 repos, split public (11) / held-out (12) / commercial (18).
| Repo | # |
|---|---|
| ansible/ansible | 96 |
| internetarchive/openlibrary | 91 |
| flipt-io/flipt | 85 |
| qutebrowser/qutebrowser | 79 |
| gravitational/teleport | 76 |
| protonmail/webclients | 65 |
| future-architect/vuls | 62 |
| navidrome/navidrome | 57 |
| element-hq/element-web | 56 |
| NodeBB/NodeBB | 44 |
| tutao/tutanota | 20 |
The held-out (12 repos) and commercial (18 repos) splits are deliberately non-public — their 1,134 instances (1,865 − 731) cannot be independently enumerated by external auditors.
- Held-out: access withheld to prevent pre-publication contamination
- Commercial: proprietary, under startup NDAs
Gold patches are large and multi-file — consistent with "hours to days" professional effort.
Every instance ships the human-augmentation fields; 100% carry resolvability context.
| Field | Coverage |
|---|---|
| problem_statement | 731 |
| requirements | 731 |
| interface | 731 |
| test_patch | 731 |
| dockerhub_tag | 731 |
The dataset README documents requirements & interface as extra fields beyond SWE-Bench Verified, fulfilling the paper's human-augmented context claim.
Context adequacy is measurable, not just asserted:
- Mean problem_statement length ≈ 1,297 chars
- Range 419–8,036 chars
Combined with the 85.6% multi-file patch rate, the public subset already exhibits the structural signature the abstract promises at the full-benchmark scale.
Spans business, B2B & dev-tools across go / js / python / ts; containerised, post-cutoff envs.
- Business: tutanota, protonmail, openlibrary, NodeBB
- B2B: flipt, teleport, navidrome, vuls
- Dev tools: ansible, element-web, qutebrowser
dockerhub_tag + base_commit; held-out/commercial non-public → no training data leakage.Loaded the official public release ScaleAI/SWE-bench_Pro and audited structure with a single HuggingFace CPU job.
- Counted instances & repos; measured patch complexity metrics
- Checked context-augmentation field coverage; validated all README fields
Yashp2003/6a5c8046… · artifacts in swebenchpro-repro-artifacts bucket. The script is deterministic and re-runs in ~3 minutes on a free CPU tier.Public instances span four stacks, confirming cross-domain enterprise coverage:
A single niche language would have been a fingerprint of a toy benchmark; four production stacks is the enterprise spread the paper advertises. The repo_language field is present on all 731 instances, so the split is verifiable, not asserted. Mixed-language codebases are where current agents still struggle most, so this spread is the point.
| Axis | This repro | Full |
|---|---|---|
| Scope | public audit | agentic solve |
| Hardware | 1x CPU | many GPUs |
| Compute | ~3 min | days |
| Cost | < 0.01 USD | 100s USD |
- Dataset: ScaleAI/SWE-bench_Pro
- Eval repo: scaleapi/SWE-bench_Pro-os
- Job: Yashp2003/6a5c8046d216bd6f3a2009d5
- Bucket: swebenchpro-repro-artifacts
- Paper: arXiv 2509.16941
| Claim | Verdict |
|---|---|
| 1 · 1,865 / 41 | partial |
| 2 · 11/12/18 | consistent |
| 3 · long-horizon | supported |
| 4 · human-verified | supported |
| 5 · contamination | supported |
"Partial" on Claim 1 reflects the locked held-out/commercial splits, not a contradiction — the public 731 / 11 is exactly as specified. Claims 2–5 are corroborated directly from observable artifact structure. The 1,865 / 41 total is taken from the paper and the Scale AI partner.