ICML
2026
REPRO
Reproducing SWE-Bench Pro: Can AI Agents Solve Long-Horizon SE Tasks?
A dataset-audit reproduction of the five headline claims (arXiv 2509.16941 / OpenReview uEVTdoAbnK).
Logbook & code
1Claims 1 & 2: Scale & Split
The paper reports 1,865 problems across 41 repos, split public (11) / held-out (12) / commercial (18).
731
verified public instances
11
verified public repos
11 + 12 + 18 = 41 — arithmetic is internally consistent. Only the 11 public repos are observable; held-out & commercial splits are locked by design.
2Public repo breakdown
| Repo | # |
|---|---|
| ansible/ansible | 96 |
| internetarchive/openlibrary | 91 |
| flipt-io/flipt | 85 |
| qutebrowser/qutebrowser | 79 |
| gravitational/teleport | 76 |
| protonmail/webclients | 65 |
| future-architect/vuls | 62 |
| navidrome/navidrome | 57 |
| element-hq/element-web | 56 |
| NodeBB/NodeBB | 44 |
| tutao/tutanota | 20 |
2bWhat we could not count
The held-out (12 repos) and commercial (18 repos) splits are deliberately non-public — their 1,134 instances (1,865 − 731) cannot be independently enumerated by external auditors.
- Held-out: access withheld to prevent pre-publication contamination
- Commercial: proprietary, under startup NDAs
The lock is what makes the benchmark contamination-resistant (Claim 5).
3Claim 3: Long-horizon
Gold patches are large and multi-file — consistent with "hours to days" professional effort.
85.6%
touch >1 file
5.08
mean files / patch
106
max files
Median 4 files; mean gold-patch size ≈ 13.3k chars — substantial cross-file modifications. The max single patch touches 106 files.
4Claim 4: Human-verified
Every instance ships the human-augmentation fields; 100% carry resolvability context.
| Field | Coverage |
|---|---|
| problem_statement | 731 |
| requirements | 731 |
| interface | 731 |
| test_patch | 731 |
| dockerhub_tag | 731 |
5Claim 5: Contamination-resistant
Spans business, B2B & dev-tools across go / js / python / ts; containerised, post-cutoff envs.
- Business: tutanota, protonmail, openlibrary, NodeBB
- B2B: flipt, teleport, navidrome, vuls
- Dev tools: ansible, element-web, qutebrowser
731 / 731 carry a pinned
dockerhub_tag + base_commit; held-out/commercial non-public → no training data leakage.6Method: How we verified
Loaded the official public release ScaleAI/SWE-bench_Pro and audited structure with a Hugging Face CPU Job.
- Counted instances & repos; measured patch complexity
- Checked context-augmentation field coverage
- Mapped domains & languages; inspected README schema
Job:
Yashp2003/6a5c8046… · artifacts in swebenchpro-repro-artifacts bucket.7Scope & Cost
| Axis | This repro | Full |
|---|---|---|
| Scope | public audit | agentic solve |
| Hardware | 1x CPU | many GPUs |
| Compute | ~3 min | days |
| Cost | <$0.01 | $100s+ |
Outcome: 5/5 claims structurally corroborated on the public subset; full 1,865/41 not directly countable (by design).
8Artifacts & Links
- Dataset: ScaleAI/SWE-bench_Pro
- Eval repo: scaleapi/SWE-bench_Pro-os
- Job: Yashp2003/6a5c8046d216bd6f3a2009d5
- Bucket: swebenchpro-repro-artifacts
- Paper: arXiv 2509.16941
Reproduction is a dataset audit, not an agentic re-solving of tasks (claims are abstract/structural).
9 Takeaways
Scale. 731 public problems / 11 repos confirmed; 1,865/41 stated.
Split. 11+12+18=41 matches the public/held-out/commercial design.
Long. 85.6% multi-file patches, median 4 files.
Verified. 100% carry requirements/interface/test_patch.