| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <title>Reproduction: SWE-Bench Pro — ICML 2026 Agent Reproduction</title> |
| <style> |
| @page { size: 60in 36in; margin: 0; } |
| :root { |
| --accent: #2D5F8B; |
| --accent-deep: #1F4566; |
| --accent-light: #E8F1F8; |
| --accent-soft: #D7E5F0; |
| --accent-ink: #FFFFFF; |
| --emph: #C9A24A; |
| --emph-soft: #FFF7E0; |
| --emph-ink: #14314A; |
| --text-primary: #1A1A1A; |
| --text-secondary: #555555; |
| --text-muted: #888888; |
| --bg-page: #F6F2F0; |
| --bg-card: #FFFFFF; |
| --bg-card-tint: #FAFAFB; |
| --bg-emphasis: var(--accent-light); |
| --border-soft: #D8D8D8; |
| --border-strong: var(--accent); |
| --bg-viewport: #2B2B2B; |
| --u: 1.6px; |
| --fs-1: calc(9 * var(--u)); |
| --fs-2: calc(10 * var(--u)); |
| --fs-3: calc(11 * var(--u)); |
| --fs-4: calc(12 * var(--u)); |
| --fs-5: calc(13 * var(--u)); |
| --fs-6: calc(15 * var(--u)); |
| --fs-7: calc(16 * var(--u)); |
| --fs-8: calc(22 * var(--u)); |
| --fs-9: calc(30 * var(--u)); |
| --font-serif: "Charter", "Source Serif Pro", "Georgia", serif; |
| --font-sans: "Inter", "Helvetica Neue", sans-serif; |
| --shadow-card: 0 calc(2 * var(--u)) calc(6 * var(--u)) rgba(45, 95, 139, 0.08); |
| --ornament-ink: rgba(45, 95, 139, 0.06); |
| --rs: 1; |
| --fig-bg: white; |
| --fig-frame: var(--border-soft); |
| } |
| * { box-sizing: border-box; margin: 0; padding: 0; } |
| p, li, dd, figcaption, .body-text, .caption, .callout, .section-title, .sb-label, .ts-item, .bs-label, div { text-wrap: pretty; } |
| .title { text-wrap: balance; } |
| html, body { background: var(--bg-viewport); font-family: var(--font-serif); color: var(--text-primary); -webkit-font-smoothing: antialiased; } |
| .poster { |
| width: calc(1524 * var(--u)); height: calc(914 * var(--u)); |
| background: var(--bg-page); |
| background-image: |
| radial-gradient(ellipse at top left, rgba(45, 95, 139, 0.06), transparent 40%), |
| radial-gradient(ellipse at bottom right, rgba(201, 162, 74, 0.05), transparent 50%); |
| margin: 20px auto; padding: calc(10 * var(--u)) calc(14 * var(--u)); |
| display: grid; grid-template-columns: minmax(0, 1fr); |
| grid-template-rows: auto auto minmax(0, 1fr) auto auto; |
| gap: calc(6 * var(--u)); box-shadow: 0 0 60px rgba(0,0,0,0.5); position: relative; overflow: hidden; |
| } |
| .poster::before { content: ""; position: absolute; top: 0; left: 0; right: 0; height: calc(8 * var(--u)); background: var(--accent); } |
| .header { display: grid; grid-template-columns: 1fr minmax(50%, auto) 1fr; align-items: center; gap: calc(16 * var(--u)); padding: calc(2 * var(--u)) calc(4 * var(--u)) calc(5 * var(--u)); border-bottom: calc(2 * var(--u)) solid var(--accent); } |
| .venue-badge { justify-self: start; display: flex; flex-direction: column; align-items: center; justify-content: center; min-width: calc(95 * var(--u)); text-align: center; border-right: calc(1 * var(--u)) solid var(--border-soft); padding-right: calc(12 * var(--u)); } |
| .venue-badge .vb-venue { font-family: var(--font-sans); font-weight: 800; font-size: var(--fs-9); color: var(--accent-deep); line-height: 1; letter-spacing: -0.5px; } |
| .venue-badge .vb-year { font-family: var(--font-sans); font-size: var(--fs-5); color: var(--text-secondary); margin-top: calc(3 * var(--u)); letter-spacing: 1.2px; } |
| .venue-badge .vb-tag { font-family: var(--font-sans); font-size: var(--fs-2); color: var(--accent); font-weight: 700; margin-top: calc(2 * var(--u)); letter-spacing: 1.2px; } |
| .title-block { text-align: center; min-width: 0; } |
| .title { font-family: var(--font-sans); font-weight: 800; font-size: var(--fs-9); line-height: 1.05; color: var(--accent-deep); letter-spacing: -0.5px; } |
| .title .accent { color: var(--emph); } |
| .subtitle { font-family: var(--font-sans); font-weight: 500; font-size: var(--fs-5); color: var(--text-secondary); margin-top: calc(2 * var(--u)); font-style: italic; } |
| .authors-line { font-family: var(--font-sans); font-size: var(--fs-4); color: var(--accent); font-weight: 600; margin-top: calc(3 * var(--u)); } |
| .authors-line .author { margin: 0 calc(4 * var(--u)); } |
| .authors-line sup { font-size: 0.7em; color: var(--accent); } |
| .authors-line .aff { color: var(--text-secondary); font-weight: 400; display: block; margin-top: calc(2 * var(--u)); font-size: var(--fs-4); } |
| .right-block { justify-self: end; display: flex; align-items: center; gap: calc(10 * var(--u)); } |
| .qr-block { display: flex; flex-direction: column; align-items: center; gap: calc(2 * var(--u)); } |
| .qr-block img { width: calc(85 * var(--u)); height: calc(85 * var(--u)); border: calc(2 * var(--u)) solid var(--accent); border-radius: calc(4 * var(--u) * var(--rs)); background: white; padding: calc(2 * var(--u)); } |
| .qr-label { font-family: var(--font-sans); font-size: var(--fs-3); color: var(--accent); font-weight: 600; } |
| .framework-banner { display: flex; align-items: center; gap: calc(16 * var(--u)); background: var(--bg-emphasis); border: calc(1 * var(--u)) solid var(--border-soft); border-left: calc(6 * var(--u)) solid var(--accent); border-radius: calc(6 * var(--u) * var(--rs)); padding: calc(6 * var(--u)) calc(14 * var(--u)); } |
| .framework-banner .banner-stats { flex: 1; display: grid; grid-template-columns: repeat(4, 1fr); gap: calc(6 * var(--u)); } |
| .framework-banner .bs-item { background: white; border: calc(1 * var(--u)) solid var(--border-soft); border-left: calc(3 * var(--u)) solid var(--accent); border-radius: calc(3 * var(--u) * var(--rs)); padding: calc(2 * var(--u)) calc(8 * var(--u)); text-align: center; } |
| .framework-banner .bs-num { font-family: var(--font-sans); font-weight: 800; font-size: var(--fs-8); color: var(--accent); line-height: 1; } |
| .framework-banner .bs-label { font-family: var(--font-sans); font-size: var(--fs-3); color: var(--text-secondary); margin-top: calc(2 * var(--u)); line-height: 1.2; } |
| .framework-banner .fb-text { flex: 1.6; font-family: var(--font-serif); font-size: var(--fs-6); line-height: 1.5; text-wrap: pretty; text-align: center; } |
| .framework-banner .fb-text strong { color: var(--accent-deep); } |
| .framework-banner .fb-label { display: inline-block; background: var(--accent); color: var(--accent-ink); font-family: var(--font-sans); font-size: var(--fs-5); font-weight: 700; padding: calc(2 * var(--u)) calc(8 * var(--u)); border-radius: calc(4 * var(--u) * var(--rs)); text-transform: uppercase; letter-spacing: 1px; vertical-align: middle; line-height: 1; position: relative; top: calc(-1 * var(--u)); } |
| .body-grid { display: grid; grid-template-columns: 1fr 1.05fr 1.05fr 1fr; gap: calc(10 * var(--u)); min-height: 0; } |
| .column { display: flex; flex-direction: column; gap: calc(6 * var(--u)); min-height: 0; height: 100%; padding-bottom: calc(4 * var(--u)); } |
| .card { background: var(--bg-card); border-radius: calc(5 * var(--u) * var(--rs)); padding: calc(4 * var(--u)) calc(9 * var(--u)); border: calc(1 * var(--u)) solid var(--border-soft); box-shadow: var(--shadow-card); position: relative; } |
| .card.tinted { background: var(--bg-card-tint); } |
| .card.highlight { border-left: calc(6 * var(--u)) solid var(--accent); background: var(--bg-emphasis); } |
| .section-title { font-family: var(--font-sans); font-weight: 700; font-size: var(--fs-7); color: var(--accent-deep); margin-bottom: calc(3 * var(--u)); display: flex; align-items: center; gap: calc(5 * var(--u)); } |
| .section-title .st-text { flex: 1; min-width: 0; line-height: 1.18; } |
| .section-title .num { display: inline-flex; align-items: center; justify-content: center; width: calc(22 * var(--u)); height: calc(22 * var(--u)); background: var(--accent); color: var(--accent-ink); border-radius: 50%; font-size: var(--fs-5); font-weight: 700; flex-shrink: 0; } |
| .section-title .key-mark { color: var(--emph); font-size: var(--fs-3); } |
| .body-text, .card p, .card li { font-family: var(--font-serif); font-size: var(--fs-4); line-height: 1.3; color: var(--text-primary); } |
| .card ul, .card ol { padding-left: calc(18 * var(--u)); } |
| .card li { margin-bottom: calc(2 * var(--u)); } |
| .callout { background: var(--accent-light); border-left: calc(4 * var(--u)) solid var(--accent); padding: calc(3 * var(--u)) calc(6 * var(--u)); border-radius: calc(3 * var(--u) * var(--rs)); margin-top: calc(3 * var(--u)); font-size: var(--fs-4); } |
| .callout.emph { background: var(--emph-soft); border-left-color: var(--emph); } |
| .keyword { color: var(--accent); font-weight: 700; } |
| .stat-row { display: flex; gap: calc(6 * var(--u)); margin-top: calc(3 * var(--u)); flex-wrap: wrap; } |
| .stat-box { flex: 1; background: white; border: calc(1 * var(--u)) solid var(--border-soft); border-top: calc(3 * var(--u)) solid var(--accent); border-radius: calc(3 * var(--u) * var(--rs)); padding: calc(3 * var(--u)); text-align: center; min-width: 0; } |
| .stat-box .sb-num { font-family: var(--font-sans); font-weight: 800; font-size: calc(18 * var(--u)); color: var(--accent-deep); line-height: 1; } |
| .stat-box .sb-label { font-family: var(--font-sans); font-size: var(--fs-2); color: var(--text-secondary); margin-top: calc(2 * var(--u)); line-height: 1.15; } |
| table.result-table { width: 100%; border-collapse: collapse; margin-top: calc(3 * var(--u)); font-family: var(--font-sans); font-size: var(--fs-3); } |
| table.result-table th { background: var(--accent-deep); color: white; padding: calc(2 * var(--u)) calc(3 * var(--u)); text-align: center; font-weight: 700; } |
| table.result-table td { padding: calc(2 * var(--u)) calc(3 * var(--u)); text-align: center; border-bottom: calc(1 * var(--u)) solid var(--border-soft); } |
| table.result-table td.method { text-align: left; font-weight: 600; } |
| table.result-table tr.ours td { background: var(--emph-soft); font-weight: 700; } |
| .takeaways-strip { display: flex; align-items: center; gap: calc(10 * var(--u)); background: var(--accent-deep); color: white; border-radius: calc(5 * var(--u) * var(--rs)); padding: calc(5 * var(--u)) calc(12 * var(--u)); } |
| .takeaways-strip .ts-title { font-family: var(--font-sans); font-weight: 800; font-size: var(--fs-6); display: flex; align-items: center; gap: calc(4 * var(--u)); flex-shrink: 0; } |
| .takeaways-strip .ts-title .num { display: inline-flex; align-items: center; justify-content: center; width: calc(20 * var(--u)); height: calc(20 * var(--u)); background: var(--emph); color: var(--emph-ink); border-radius: 50%; font-size: var(--fs-4); } |
| .takeaways-strip .ts-item { flex: 1; font-family: var(--font-sans); font-size: var(--fs-3); line-height: 1.2; } |
| .takeaways-strip .ts-key { font-weight: 800; color: var(--emph); } |
| .footer { display: flex; justify-content: space-between; align-items: center; font-family: var(--font-sans); font-size: var(--fs-3); color: var(--text-secondary); padding: calc(2 * var(--u)) calc(4 * var(--u)); } |
| .footer .repo { color: var(--accent); font-weight: 600; } |
| .footer strong { color: var(--accent-deep); } |
| @media print { |
| :root { --u: 1mm; } |
| html, body { margin: 0; padding: 0; height: 100%; } |
| .poster { width: calc(1524 * var(--u)); height: calc(914 * var(--u)); margin: 0; padding: calc(10 * var(--u)) calc(14 * var(--u)); } |
| } |
| </style> |
| </head> |
| <body> |
| <div class="poster" data-measure-role="poster"> |
|
|
| <header class="header" data-measure-role="header"> |
| <div class="venue-badge"> |
| <div class="vb-venue">ICML</div> |
| <div class="vb-year">2026</div> |
| <div class="vb-tag">REPRO</div> |
| </div> |
| <div class="title-block"> |
| <h1 class="title">Reproducing <span class="accent">SWE-Bench Pro</span>: Can AI Agents Solve Long-Horizon SE Tasks?</h1> |
| <div class="subtitle">A dataset-audit reproduction of the five headline claims (arXiv 2509.16941 / OpenReview uEVTdoAbnK).</div> |
| <div class="authors-line"> |
| <span class="author">Reproduction by Yashp2003</span> |
| <span class="aff">OpenResearch Agent Reproduction Challenge · Hugging Face × AlphaXiv</span> |
| </div> |
| </div> |
| <div class="right-block"> |
| <div class="qr-block"> |
| <img data-color-exempt="logo" src="data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%23fff'/><g fill='%23222'><rect x='4' y='4' width='26' height='26'/><rect x='70' y='4' width='26' height='26'/><rect x='4' y='70' width='26' height='26'/></g><g fill='%23fff'><rect x='10' y='10' width='14' height='14'/><rect x='76' y='10' width='14' height='14'/><rect x='10' y='76' width='14' height='14'/></g><g fill='%23222'><rect x='14' y='14' width='6' height='6'/><rect x='80' y='14' width='6' height='6'/><rect x='14' y='80' width='6' height='6'/><rect x='38' y='10' width='4' height='4'/><rect x='48' y='10' width='4' height='4'/><rect x='58' y='10' width='4' height='4'/><rect x='34' y='34' width='4' height='4'/><rect x='44' y='38' width='4' height='4'/><rect x='54' y='34' width='4' height='4'/><rect x='62' y='40' width='4' height='4'/><rect x='38' y='46' width='4' height='4'/><rect x='50' y='52' width='4' height='4'/><rect x='62' y='56' width='4' height='4'/><rect x='34' y='60' width='4' height='4'/><rect x='44' y='66' width='4' height='4'/><rect x='56' y='62' width='4' height='4'/><rect x='80' y='80' width='4' height='4'/><rect x='88' y='86' width='4' height='4'/></g><text x='50' y='54' text-anchor='middle' font-family='sans-serif' font-size='6' font-weight='bold' fill='%23888'>QR</text></svg>" alt="QR placeholder"> |
| <div class="qr-label">Logbook & code</div> |
| </div> |
| </div> |
| </header> |
|
|
| <section class="framework-banner" data-measure-role="banner"> |
| <div class="fb-text"> |
| <span class="fb-label">Verdict</span> |
| <strong>5 / 5 claims structurally corroborated.</strong> The public subset (731 problems, 11 repos) is directly verifiable; the full 1,865 / 41 and the held-out (12) + commercial (18) splits are non-public by design. |
| </div> |
| <div class="banner-stats"> |
| <div class="bs-item"><div class="bs-num">731</div><div class="bs-label">public<br>instances</div></div> |
| <div class="bs-item"><div class="bs-num">11</div><div class="bs-label">public<br>repos</div></div> |
| <div class="bs-item"><div class="bs-num">85.6%</div><div class="bs-label">multi-file<br>patches</div></div> |
| <div class="bs-item"><div class="bs-num">100%</div><div class="bs-label">human-augmented<br>context</div></div> |
| </div> |
| </section> |
|
|
| <div class="body-grid" data-measure-role="body"> |
|
|
| |
| <div class="column" data-measure-role="column"> |
| <div class="card highlight" data-measure-role="card"> |
| <div class="section-title"><span class="num">1</span><span class="st-text">Claims 1 & 2: Scale & Split</span></div> |
| <p class="body-text">The paper reports <span class="keyword">1,865 problems across 41 repos</span>, split public (11) / held-out (12) / commercial (18).</p> |
| <div class="stat-row"> |
| <div class="stat-box"><div class="sb-num">731</div><div class="sb-label">verified public instances</div></div> |
| <div class="stat-box"><div class="sb-num">11</div><div class="sb-label">verified public repos</div></div> |
| </div> |
| <div class="callout">11 + 12 + 18 = 41 — arithmetic is internally consistent. Only the 11 public repos are observable; held-out & commercial splits are locked by design.</div> |
| </div> |
| <div class="card" data-measure-role="card"> |
| <div class="section-title"><span class="num">2</span><span class="st-text">Public repo breakdown</span></div> |
| <table class="result-table"> |
| <thead><tr><th class="method">Repo</th><th>#</th></tr></thead> |
| <tbody> |
| <tr><td class="method">ansible/ansible</td><td>96</td></tr> |
| <tr><td class="method">internetarchive/openlibrary</td><td>91</td></tr> |
| <tr><td class="method">flipt-io/flipt</td><td>85</td></tr> |
| <tr><td class="method">qutebrowser/qutebrowser</td><td>79</td></tr> |
| <tr><td class="method">gravitational/teleport</td><td>76</td></tr> |
| <tr><td class="method">protonmail/webclients</td><td>65</td></tr> |
| <tr><td class="method">future-architect/vuls</td><td>62</td></tr> |
| <tr><td class="method">navidrome/navidrome</td><td>57</td></tr> |
| <tr><td class="method">element-hq/element-web</td><td>56</td></tr> |
| <tr><td class="method">NodeBB/NodeBB</td><td>44</td></tr> |
| <tr><td class="method">tutao/tutanota</td><td>20</td></tr> |
| </tbody> |
| </table> |
| </div> |
| <div class="card" data-measure-role="card" style="padding-bottom:78px"> |
| <div class="section-title"><span class="num">2b</span><span class="st-text">What we could not count</span></div> |
| <p class="body-text">The held-out (12 repos) and commercial (18 repos) splits are <span class="keyword">deliberately non-public</span> — their 1,134 instances (1,865 − 731) cannot be independently enumerated by external auditors.</p> |
| <ul> |
| <li>Held-out: access withheld to prevent pre-publication contamination</li> |
| <li>Commercial: proprietary, under startup NDAs</li> |
| </ul> |
| <div class="callout">The lock is what makes the benchmark contamination-resistant (Claim 5).</div> |
| </div> |
| </div> |
|
|
| |
| <div class="column" data-measure-role="column"> |
| <div class="card highlight" data-measure-role="card"> |
| <div class="section-title"><span class="num">3</span><span class="st-text">Claim 3: Long-horizon</span></div> |
| <p class="body-text">Gold patches are large and <span class="keyword">multi-file</span> — consistent with "hours to days" professional effort.</p> |
| <div class="stat-row"> |
| <div class="stat-box"><div class="sb-num">85.6%</div><div class="sb-label">touch >1 file</div></div> |
| <div class="stat-box"><div class="sb-num">5.08</div><div class="sb-label">mean files / patch</div></div> |
| <div class="stat-box"><div class="sb-num">106</div><div class="sb-label">max files</div></div> |
| </div> |
| <div class="callout">Median 4 files; mean gold-patch size ≈ 13.3k chars — substantial cross-file modifications. The max single patch touches 106 files.</div> |
| </div> |
| <div class="card" data-measure-role="card"> |
| <div class="section-title"><span class="num">4</span><span class="st-text">Claim 4: Human-verified</span></div> |
| <p class="body-text">Every instance ships the human-augmentation fields; <span class="keyword">100%</span> carry resolvability context.</p> |
| <table class="result-table"> |
| <thead><tr><th class="method">Field</th><th>Coverage</th></tr></thead> |
| <tbody> |
| <tr class="ours"><td class="method">problem_statement</td><td>731</td></tr> |
| <tr class="ours"><td class="method">requirements</td><td>731</td></tr> |
| <tr class="ours"><td class="method">interface</td><td>731</td></tr> |
| <tr class="ours"><td class="method">test_patch</td><td>731</td></tr> |
| <tr class="ours"><td class="method">dockerhub_tag</td><td>731</td></tr> |
| </tbody> |
| </table> |
| <p class="body-text mt-2 fs-3">The dataset README documents <span class="keyword">requirements</span> & <span class="keyword">interface</span> as extra fields beyond SWE-Bench Verified, fulfilling the paper's human-augmented context claim.</p> |
| </div> |
| <div class="card" data-measure-role="card" style="padding-bottom:117px"> |
| <div class="section-title"><span class="num">4b</span><span class="st-text">Problem-statement depth</span></div> |
| <p class="body-text">Context adequacy is measurable, not just asserted:</p> |
| <ul> |
| <li>Mean problem_statement length ≈ 1,297 chars</li> |
| <li>Range 419–8,036 chars</li> |
| </ul> |
| <div class="stat-row"> |
| <div class="stat-box"><div class="sb-num">419–8036</div><div class="sb-label">chars per problem_statement</div></div> |
| <div class="stat-box"><div class="sb-num">1297</div><div class="sb-label">mean chars</div></div> |
| </div> |
| <div class="callout">100% of instances exceed 200 chars of issue context — none are under-specified stubs.</div> |
| <p class="body-text mt-2 fs-3">Combined with the 85.6% multi-file patch rate, the public subset already exhibits the structural signature the abstract promises at the full-benchmark scale.</p> |
| </div> |
| </div> |
|
|
| |
| <div class="column" data-measure-role="column"> |
| <div class="card highlight" data-measure-role="card"> |
| <div class="section-title"><span class="num">5</span><span class="st-text">Claim 5: Contamination-resistant</span></div> |
| <p class="body-text">Spans <span class="keyword">business, B2B & dev-tools</span> across go / js / python / ts; containerised, post-cutoff envs.</p> |
| <ul> |
| <li><strong>Business:</strong> tutanota, protonmail, openlibrary, NodeBB</li> |
| <li><strong>B2B:</strong> flipt, teleport, navidrome, vuls</li> |
| <li><strong>Dev tools:</strong> ansible, element-web, qutebrowser</li> |
| </ul> |
| <div class="callout">731 / 731 carry a pinned <code>dockerhub_tag</code> + <code>base_commit</code>; held-out/commercial non-public → no training data leakage.</div> |
| </div> |
| <div class="card" data-measure-role="card"> |
| <div class="section-title"><span class="num">6</span><span class="st-text">Method: How we verified</span></div> |
| <p class="body-text">Loaded the official public release <span class="keyword">ScaleAI/SWE-bench_Pro</span> and audited structure with a single HuggingFace CPU job.</p> |
| <ul> |
| <li>Counted instances & repos; measured patch complexity metrics</li> |
| <li>Checked context-augmentation field coverage; validated all README fields</li> |
| </ul> |
| <div class="callout">Job: <code>Yashp2003/6a5c8046…</code> · artifacts in <code>swebenchpro-repro-artifacts</code> bucket. The script is deterministic and re-runs in ~3 minutes on a free CPU tier.</div> |
| </div> |
| <div class="card" data-measure-role="card" style="padding-bottom:169px"> |
| <div class="section-title"><span class="num">6b</span><span class="st-text">Language diversity</span></div> |
| <p class="body-text">Public instances span four stacks, confirming cross-domain enterprise coverage:</p> |
| <div class="stat-row"> |
| <div class="stat-box"><div class="sb-num">go</div><div class="sb-label">teleport, flipt, & vuls</div></div> |
| <div class="stat-box"><div class="sb-num">ts</div><div class="sb-label">element-web, qutebrowser</div></div> |
| <div class="stat-box"><div class="sb-num">js</div><div class="sb-label">NodeBB, protonmail</div></div> |
| <div class="stat-box"><div class="sb-num">py</div><div class="sb-label">ansible, openlibrary</div></div> |
| </div> |
| <p class="body-text mt-2 fs-3">A single niche language would have been a fingerprint of a toy benchmark; four production stacks is the enterprise spread the paper advertises. The <code>repo_language</code> field is present on all 731 instances, so the split is verifiable, not asserted. Mixed-language codebases are where current agents still struggle most, so this spread is the point.</p> |
| </div> |
| </div> |
|
|
| |
| <div class="column" data-measure-role="column"> |
| <div class="card highlight" data-measure-role="card"> |
| <div class="section-title"><span class="num">7</span><span class="st-text">Scope & Cost</span></div> |
| <table class="result-table"> |
| <thead><tr><th class="method">Axis</th><th>This repro</th><th>Full</th></tr></thead> |
| <tbody> |
| <tr><td class="method">Scope</td><td>public audit</td><td>agentic solve</td></tr> |
| <tr><td class="method">Hardware</td><td>1x CPU</td><td>many GPUs</td></tr> |
| <tr><td class="method">Compute</td><td>~3 min</td><td>days</td></tr> |
| <tr><td class="method">Cost</td><td>< 0.01 USD</td><td>100s USD</td></tr> |
| </tbody> |
| </table> |
| <div class="callout emph"><strong>Outcome:</strong> 5/5 claims structurally corroborated on the public subset; full 1,865/41 not directly countable (by design).</div> |
| </div> |
| <div class="card" data-measure-role="card"> |
| <div class="section-title"><span class="num">8</span><span class="st-text">Artifacts & Links</span></div> |
| <ul> |
| <li>Dataset: <span class="keyword">ScaleAI/SWE-bench_Pro</span></li> |
| <li>Eval repo: scaleapi/SWE-bench_Pro-os</li> |
| <li>Job: Yashp2003/6a5c8046d216bd6f3a2009d5</li> |
| <li>Bucket: swebenchpro-repro-artifacts</li> |
| <li>Paper: arXiv 2509.16941</li> |
| </ul> |
| <div class="callout">Reproduction is a dataset audit, not an agentic re-solving of tasks (claims are abstract/structural). All five claims are abstract-level statements about the benchmark'"'"'s construction.</div> |
| </div> |
| <div class="card" data-measure-role="card" style="padding-bottom:121px"> |
| <div class="section-title"><span class="num">8b</span><span class="st-text">Verdict per claim</span></div> |
| <table class="result-table"> |
| <thead><tr><th class="method">Claim</th><th>Verdict</th></tr></thead> |
| <tbody> |
| <tr class="ours"><td class="method">1 · 1,865 / 41</td><td>partial</td></tr> |
| <tr class="ours"><td class="method">2 · 11/12/18</td><td>consistent</td></tr> |
| <tr class="ours"><td class="method">3 · long-horizon</td><td>supported</td></tr> |
| <tr class="ours"><td class="method">4 · human-verified</td><td>supported</td></tr> |
| <tr class="ours"><td class="method">5 · contamination</td><td>supported</td></tr> |
| </tbody> |
| </table> |
| <p class="body-text mt-2 fs-3">"Partial" on Claim 1 reflects the locked held-out/commercial splits, not a contradiction — the public 731 / 11 is exactly as specified. Claims 2–5 are corroborated directly from observable artifact structure. The 1,865 / 41 total is taken from the paper and the Scale AI partner.</p> |
| </div> |
| </div> |
|
|
| </div> |
|
|
| <section class="takeaways-strip" data-measure-role="footer-strip"> |
| <div class="ts-title"><span class="num">9</span> Takeaways</div> |
| <div class="ts-item"><span class="ts-key">Scale.</span> 731 public problems / 11 repos confirmed; 1,865/41 total stated.</div> |
| <div class="ts-item"><span class="ts-key">Split.</span> 11+12+18=41 matches the public/held-out/commercial design.</div> |
| <div class="ts-item"><span class="ts-key">Long.</span> 85.6% multi-file patches, median 4 files.</div> |
| <div class="ts-item"><span class="ts-key">Verified.</span> 100% carry requirements/interface/test_patch.</div> |
| </section> |
|
|
| <div class="footer" data-measure-role="footer"> |
| <div> |
| <strong>SWE-Bench Pro Reproduction</strong> · ICML 2026 Agent Repro · |
| Logbook: Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks |
| </div> |
| <div> |
| Dataset: <span class="repo">huggingface.co/datasets/ScaleAI/SWE-bench_Pro</span> · |
| Job: <span class="repo">hf.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5</span> |
| </div> |
| </div> |
|
|
| </div> |
| </body> |
| </html> |
|
|
