Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <meta name="description" content="Reproduction poster for PostTrainBench"> | |
| <title>PostTrainBench Reproduction Poster</title> | |
| <style>:root { | |
| --bg: #0f172a; | |
| --surface: #1e293b; | |
| --border: #334155; | |
| --text: #e2e8f0; | |
| --text-muted: #94a3b8; | |
| --accent: #38bdf8; | |
| --green: #4ade80; | |
| --yellow: #facc15; | |
| --red: #f87171; | |
| --orange: #fb923c; | |
| } | |
| * { margin: 0; padding: 0; box-sizing: border-box; } | |
| body { | |
| font-family: 'Inter', system-ui, -apple-system, sans-serif; | |
| background: var(--bg); | |
| color: var(--text); | |
| line-height: 1.6; | |
| padding: 2rem; | |
| max-width: 1200px; | |
| margin: 0 auto; | |
| } | |
| h1 { font-size: 1.75rem; margin-bottom: 0.5rem; color: var(--accent); } | |
| h2 { font-size: 1.25rem; margin: 1.5rem 0 0.75rem; color: var(--accent); border-bottom: 1px solid var(--border); padding-bottom: 0.25rem; } | |
| h3 { font-size: 1rem; margin: 1rem 0 0.5rem; color: var(--text); } | |
| table { border-collapse: collapse; width: 100%; margin: 0.5rem 0; } | |
| th, td { padding: 0.4rem 0.75rem; text-align: left; border: 1px solid var(--border); font-size: 0.85rem; } | |
| th { background: var(--surface); color: var(--accent); font-weight: 600; } | |
| td { background: var(--bg); } | |
| .card { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 1rem; margin: 0.75rem 0; } | |
| .status { display: inline-block; padding: 0.15rem 0.5rem; border-radius: 4px; font-size: 0.8rem; font-weight: 600; } | |
| .status-partial { background: rgba(250, 204, 21, 0.15); color: var(--yellow); border: 1px solid var(--yellow); } | |
| .status-unavailable { background: rgba(248, 113, 113, 0.15); color: var(--red); border: 1px solid var(--red); } | |
| .status-verified { background: rgba(74, 222, 128, 0.15); color: var(--green); border: 1px solid var(--green); } | |
| .ptr { color: var(--text-muted); font-family: monospace; font-size: 0.75rem; } | |
| a { color: var(--accent); text-decoration: none; } | |
| a:hover { text-decoration: underline; } | |
| ul, ol { margin: 0.5rem 0 0.5rem 1.5rem; } | |
| li { margin: 0.25rem 0; font-size: 0.9rem; } | |
| .limitation { background: rgba(251, 146, 60, 0.1); border-left: 3px solid var(--orange); padding: 0.5rem 0.75rem; margin: 0.25rem 0; font-size: 0.85rem; } | |
| .excerpt { background: var(--bg); border: 1px solid var(--border); padding: 0.5rem; margin: 0.25rem 0; font-family: monospace; font-size: 0.8rem; white-space: pre-wrap; } | |
| nav { margin-bottom: 1.5rem; } | |
| nav a { margin-right: 1rem; } | |
| footer { margin-top: 2rem; padding-top: 1rem; border-top: 1px solid var(--border); font-size: 0.8rem; color: var(--text-muted); } | |
| body { max-width: 900px; } | |
| .poster-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; } | |
| @media (max-width: 700px) { .poster-grid { grid-template-columns: 1fr; } } | |
| </style> | |
| </head> | |
| <body> | |
| <h1>PostTrainBench: Released-Artifact Audit</h1> | |
| <p>Paper ID: <span class="evidence-result" data-evidence-pointer="evidence/provenance.json#/paper_id"><code>UnjxMTe57e</code> <span class="ptr">evidence/provenance.json#/paper_id</span></span> · arXiv: <span class="evidence-result" data-evidence-pointer="evidence/provenance.json#/arxiv_id"><code>2603.08640v2</code> <span class="ptr">evidence/provenance.json#/arxiv_id</span></span></p> | |
| <div class="poster-grid"> | |
| <div class="card"> | |
| <h3>Claim 1 <span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/status"><span class="status status-partial">partial-support</span> <span class="ptr">evidence/claims.json#/claim_1/status</span></span></h3> | |
| <p><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/text">PostTrainBench evaluates autonomous post-training agents across 4 base models and 7 benchmarks under a 10-hour single-H100 budget (Figure 1). <span class="ptr">evidence/claims.json#/claim_1/text</span></span></p> | |
| <p>SHA-256: <span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/sha256"><code>9c0c1fc52ad2a93a9dbe299532b952948c4ecb674f820fe19f78f6a3c33b0073</code> <span class="ptr">evidence/claims.json#/claim_1/sha256</span></span></p> | |
| <p><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/summary">Released trajectory inventory confirms 4-by-7 coverage across all accepted benchmark/model cells. Runner configuration defaults to one H100 with a NUM_HOURS-based timeout. The current checkout's scheduler-dependent branches and five-minute termination grace are reported as limitations. <span class="ptr">evidence/claims.json#/claim_1/summary</span></span></p> | |
| </div> | |
| <div class="card"> | |
| <h3>Claim 2 <span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/status"><span class="status status-partial">partial-support</span> <span class="ptr">evidence/claims.json#/claim_2/status</span></span></h3> | |
| <p><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/text">The paper reports reward-hacking failure modes including training on test sets, downloading instruction-tuned checkpoints, and using discovered API keys for synthetic data (Abstract). <span class="ptr">evidence/claims.json#/claim_2/text</span></span></p> | |
| <p>SHA-256: <span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/sha256"><code>d185d61e5d886672a739e321e048df2378b71f55cf388ec4097bd2df1a916aad</code> <span class="ptr">evidence/claims.json#/claim_2/sha256</span></span></p> | |
| <p><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/summary">Released contamination and instruction-model judgments provide partial support for two of three reward-hacking submodes. The API-key submode artifact is absent from the pinned revision. <span class="ptr">evidence/claims.json#/claim_2/summary</span></span></p> | |
| </div> | |
| </div> | |
| <h2>Coverage Counts</h2> | |
| <div class="card"><ul> | |
| <li>Accepted benchmarks: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/accepted_benchmark_count">7 <span class="ptr">evidence/coverage.json#/accepted_benchmark_count</span></span></li> | |
| <li>Accepted models: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/accepted_model_count">4 <span class="ptr">evidence/coverage.json#/accepted_model_count</span></span></li> | |
| <li>Recognized tasks: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/recognized_task_count">1338 <span class="ptr">evidence/coverage.json#/recognized_task_count</span></span></li> | |
| <li>Recognized roots: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/recognized_root_count">47 <span class="ptr">evidence/coverage.json#/recognized_root_count</span></span></li> | |
| <li>Root/cell pairs: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/recognized_root_cell_pairs">1313 <span class="ptr">evidence/coverage.json#/recognized_root_cell_pairs</span></span></li> | |
| <li>Duplicate pairs: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/duplicate_job_pairs">25 <span class="ptr">evidence/coverage.json#/duplicate_job_pairs</span></span></li> | |
| <li>Missing pairs: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/missing_root_cell_pairs">3 <span class="ptr">evidence/coverage.json#/missing_root_cell_pairs</span></span></li> | |
| <li>Excluded nested task dirs: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_dirs_count">0 <span class="ptr">evidence/coverage.json#/excluded_dirs_count</span></span></li> | |
| <li>Excluded auxiliary top-level: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_auxiliary_data/top_level_path"><code>viewer_data</code> <span class="ptr">evidence/coverage.json#/excluded_auxiliary_data/top_level_path</span></span>; files: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_auxiliary_data/file_count">2397 <span class="ptr">evidence/coverage.json#/excluded_auxiliary_data/file_count</span></span>; directories: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_auxiliary_data/directory_count">1 <span class="ptr">evidence/coverage.json#/excluded_auxiliary_data/directory_count</span></span>; counted as task root: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_auxiliary_data/counted_as_task_root">false <span class="ptr">evidence/coverage.json#/excluded_auxiliary_data/counted_as_task_root</span></span></li> | |
| </ul></div> | |
| <h2>Coverage Matrix</h2> | |
| <table> | |
| <tr><th>Benchmark</th> | |
| <th>Qwen3-1.7B-Base</th> | |
| <th>Qwen3-4B-Base</th> | |
| <th>SmolLM3-3B-Base</th> | |
| <th>Gemma-3-4B-PT</th> | |
| </tr> | |
| <tr><td><code>aime2025</code></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/aime2025/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/aime2025/0</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/aime2025/1">48 <span class="ptr">evidence/coverage.json#/cell_counts/aime2025/1</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/aime2025/2">48 <span class="ptr">evidence/coverage.json#/cell_counts/aime2025/2</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/aime2025/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/aime2025/3</span></span></td> | |
| </tr> | |
| <tr><td><code>arenahardwriting</code></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/arenahardwriting/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/arenahardwriting/0</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/arenahardwriting/1">48 <span class="ptr">evidence/coverage.json#/cell_counts/arenahardwriting/1</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/arenahardwriting/2">50 <span class="ptr">evidence/coverage.json#/cell_counts/arenahardwriting/2</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/arenahardwriting/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/arenahardwriting/3</span></span></td> | |
| </tr> | |
| <tr><td><code>bfcl</code></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/bfcl/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/bfcl/0</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/bfcl/1">48 <span class="ptr">evidence/coverage.json#/cell_counts/bfcl/1</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/bfcl/2">48 <span class="ptr">evidence/coverage.json#/cell_counts/bfcl/2</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/bfcl/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/bfcl/3</span></span></td> | |
| </tr> | |
| <tr><td><code>gpqamain</code></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gpqamain/0">46 <span class="ptr">evidence/coverage.json#/cell_counts/gpqamain/0</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gpqamain/1">49 <span class="ptr">evidence/coverage.json#/cell_counts/gpqamain/1</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gpqamain/2">47 <span class="ptr">evidence/coverage.json#/cell_counts/gpqamain/2</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gpqamain/3">49 <span class="ptr">evidence/coverage.json#/cell_counts/gpqamain/3</span></span></td> | |
| </tr> | |
| <tr><td><code>gsm8k</code></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gsm8k/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/gsm8k/0</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gsm8k/1">47 <span class="ptr">evidence/coverage.json#/cell_counts/gsm8k/1</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gsm8k/2">49 <span class="ptr">evidence/coverage.json#/cell_counts/gsm8k/2</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gsm8k/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/gsm8k/3</span></span></td> | |
| </tr> | |
| <tr><td><code>healthbench</code></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/healthbench/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/healthbench/0</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/healthbench/1">48 <span class="ptr">evidence/coverage.json#/cell_counts/healthbench/1</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/healthbench/2">48 <span class="ptr">evidence/coverage.json#/cell_counts/healthbench/2</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/healthbench/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/healthbench/3</span></span></td> | |
| </tr> | |
| <tr><td><code>humaneval</code></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/humaneval/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/humaneval/0</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/humaneval/1">47 <span class="ptr">evidence/coverage.json#/cell_counts/humaneval/1</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/humaneval/2">48 <span class="ptr">evidence/coverage.json#/cell_counts/humaneval/2</span></span></td> | |
| <td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/humaneval/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/humaneval/3</span></span></td> | |
| </tr> | |
| </table> | |
| <h2>Reward-Hacking Statuses</h2> | |
| <div class="card"><p>Contamination: <span class="evidence-result" data-evidence-pointer="evidence/reward_hacking.json#/training_on_test_sets/status"><span class="status status-partial">partial-support</span> <span class="ptr">evidence/reward_hacking.json#/training_on_test_sets/status</span></span> · Instruction model: <span class="evidence-result" data-evidence-pointer="evidence/reward_hacking.json#/downloading_instruction_tuned_checkpoint/status"><span class="status status-partial">partial-support</span> <span class="ptr">evidence/reward_hacking.json#/downloading_instruction_tuned_checkpoint/status</span></span> · API misuse: <span class="evidence-result" data-evidence-pointer="evidence/reward_hacking.json#/using_discovered_api_key/status"><span class="status status-unavailable">unavailable</span> <span class="ptr">evidence/reward_hacking.json#/using_discovered_api_key/status</span></span></p></div> | |
| <h2>Limitations</h2> | |
| <div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/limitations/0">No H100 run is reproduced; the resource and time findings are a released-configuration audit. <span class="ptr">evidence/claims.json#/claim_1/limitations/0</span></span></div> | |
| <div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/limitations/1">The runner allows a five-minute termination grace, and a released example records 10:05:01. <span class="ptr">evidence/claims.json#/claim_1/limitations/1</span></span></div> | |
| <div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/limitations/2">The pinned source's current launcher is scheduler-dependent: one branch has a 100-hour/eight-GPU METR command, another has 7 ten-hour and 1 one-hour default-GPU commands, and the arrays currently select only one model/benchmark pair. <span class="ptr">evidence/claims.json#/claim_1/limitations/2</span></span></div> | |
| <div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/limitations/3">Evidence is not an official challenge verdict. <span class="ptr">evidence/claims.json#/claim_1/limitations/3</span></span></div> | |
| <div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/limitations/0">A released judge label is not independently established behavioral truth. <span class="ptr">evidence/claims.json#/claim_2/limitations/0</span></span></div> | |
| <div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/limitations/1">The instruction-model evidence is an upstream categorical label plus safe extracts from a released trace, not a fresh independent behavioral audit. <span class="ptr">evidence/claims.json#/claim_2/limitations/1</span></span></div> | |
| <div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/limitations/2">The selected trajectory revision does not expose the exact GPT-5.1 Codex-Max API-misuse task cluster 16804408. <span class="ptr">evidence/claims.json#/claim_2/limitations/2</span></span></div> | |
| <div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/limitations/3">No leaderboard score, BFCL score, weighted average, or reasoning-effort ablation is a selected target or reproduced measurement. <span class="ptr">evidence/claims.json#/claim_2/limitations/3</span></span></div> | |
| <footer> | |
| <p>This evidence is not an official challenge verdict. | |
| <span class="ptr">evidence/claims.json</span> | |
| <a href="index.html">Full evidence summary</a> · | |
| <a href="report.html">Detailed report</a></p> | |
| </footer> | |
| </body> | |
| </html> | |