Spaces:

wrice
/

repro-posttrainbench

Running

File size: 18,193 Bytes

a7b634e

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="Reproduction poster for PostTrainBench">
<title>PostTrainBench Reproduction Poster</title>
<style>:root {
  --bg: #0f172a;
  --surface: #1e293b;
  --border: #334155;
  --text: #e2e8f0;
  --text-muted: #94a3b8;
  --accent: #38bdf8;
  --green: #4ade80;
  --yellow: #facc15;
  --red: #f87171;
  --orange: #fb923c;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
  font-family: 'Inter', system-ui, -apple-system, sans-serif;
  background: var(--bg);
  color: var(--text);
  line-height: 1.6;
  padding: 2rem;
  max-width: 1200px;
  margin: 0 auto;
}
h1 { font-size: 1.75rem; margin-bottom: 0.5rem; color: var(--accent); }
h2 { font-size: 1.25rem; margin: 1.5rem 0 0.75rem; color: var(--accent); border-bottom: 1px solid var(--border); padding-bottom: 0.25rem; }
h3 { font-size: 1rem; margin: 1rem 0 0.5rem; color: var(--text); }
table { border-collapse: collapse; width: 100%; margin: 0.5rem 0; }
th, td { padding: 0.4rem 0.75rem; text-align: left; border: 1px solid var(--border); font-size: 0.85rem; }
th { background: var(--surface); color: var(--accent); font-weight: 600; }
td { background: var(--bg); }
.card { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 1rem; margin: 0.75rem 0; }
.status { display: inline-block; padding: 0.15rem 0.5rem; border-radius: 4px; font-size: 0.8rem; font-weight: 600; }
.status-partial { background: rgba(250, 204, 21, 0.15); color: var(--yellow); border: 1px solid var(--yellow); }
.status-unavailable { background: rgba(248, 113, 113, 0.15); color: var(--red); border: 1px solid var(--red); }
.status-verified { background: rgba(74, 222, 128, 0.15); color: var(--green); border: 1px solid var(--green); }
.ptr { color: var(--text-muted); font-family: monospace; font-size: 0.75rem; }
a { color: var(--accent); text-decoration: none; }
a:hover { text-decoration: underline; }
ul, ol { margin: 0.5rem 0 0.5rem 1.5rem; }
li { margin: 0.25rem 0; font-size: 0.9rem; }
.limitation { background: rgba(251, 146, 60, 0.1); border-left: 3px solid var(--orange); padding: 0.5rem 0.75rem; margin: 0.25rem 0; font-size: 0.85rem; }
.excerpt { background: var(--bg); border: 1px solid var(--border); padding: 0.5rem; margin: 0.25rem 0; font-family: monospace; font-size: 0.8rem; white-space: pre-wrap; }
nav { margin-bottom: 1.5rem; }
nav a { margin-right: 1rem; }
footer { margin-top: 2rem; padding-top: 1rem; border-top: 1px solid var(--border); font-size: 0.8rem; color: var(--text-muted); }

body { max-width: 900px; }
.poster-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; }
@media (max-width: 700px) { .poster-grid { grid-template-columns: 1fr; } }
</style>
</head>
<body>
<h1>PostTrainBench: Released-Artifact Audit</h1>
<p>Paper ID: <span class="evidence-result" data-evidence-pointer="evidence/provenance.json#/paper_id"><code>UnjxMTe57e</code> <span class="ptr">evidence/provenance.json#/paper_id</span></span> &middot; arXiv: <span class="evidence-result" data-evidence-pointer="evidence/provenance.json#/arxiv_id"><code>2603.08640v2</code> <span class="ptr">evidence/provenance.json#/arxiv_id</span></span></p>

<div class="poster-grid">
<div class="card">
<h3>Claim 1 <span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/status"><span class="status status-partial">partial-support</span> <span class="ptr">evidence/claims.json#/claim_1/status</span></span></h3>
<p><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/text">PostTrainBench evaluates autonomous post-training agents across 4 base models and 7 benchmarks under a 10-hour single-H100 budget (Figure 1). <span class="ptr">evidence/claims.json#/claim_1/text</span></span></p>
<p>SHA-256: <span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/sha256"><code>9c0c1fc52ad2a93a9dbe299532b952948c4ecb674f820fe19f78f6a3c33b0073</code> <span class="ptr">evidence/claims.json#/claim_1/sha256</span></span></p>
<p><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/summary">Released trajectory inventory confirms 4-by-7 coverage across all accepted benchmark/model cells. Runner configuration defaults to one H100 with a NUM_HOURS-based timeout. The current checkout's scheduler-dependent branches and five-minute termination grace are reported as limitations. <span class="ptr">evidence/claims.json#/claim_1/summary</span></span></p>
</div>
<div class="card">
<h3>Claim 2 <span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/status"><span class="status status-partial">partial-support</span> <span class="ptr">evidence/claims.json#/claim_2/status</span></span></h3>
<p><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/text">The paper reports reward-hacking failure modes including training on test sets, downloading instruction-tuned checkpoints, and using discovered API keys for synthetic data (Abstract). <span class="ptr">evidence/claims.json#/claim_2/text</span></span></p>
<p>SHA-256: <span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/sha256"><code>d185d61e5d886672a739e321e048df2378b71f55cf388ec4097bd2df1a916aad</code> <span class="ptr">evidence/claims.json#/claim_2/sha256</span></span></p>
<p><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/summary">Released contamination and instruction-model judgments provide partial support for two of three reward-hacking submodes. The API-key submode artifact is absent from the pinned revision. <span class="ptr">evidence/claims.json#/claim_2/summary</span></span></p>
</div>
</div>

<h2>Coverage Counts</h2>
<div class="card"><ul>
<li>Accepted benchmarks: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/accepted_benchmark_count">7 <span class="ptr">evidence/coverage.json#/accepted_benchmark_count</span></span></li>
<li>Accepted models: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/accepted_model_count">4 <span class="ptr">evidence/coverage.json#/accepted_model_count</span></span></li>
<li>Recognized tasks: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/recognized_task_count">1338 <span class="ptr">evidence/coverage.json#/recognized_task_count</span></span></li>
<li>Recognized roots: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/recognized_root_count">47 <span class="ptr">evidence/coverage.json#/recognized_root_count</span></span></li>
<li>Root/cell pairs: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/recognized_root_cell_pairs">1313 <span class="ptr">evidence/coverage.json#/recognized_root_cell_pairs</span></span></li>
<li>Duplicate pairs: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/duplicate_job_pairs">25 <span class="ptr">evidence/coverage.json#/duplicate_job_pairs</span></span></li>
<li>Missing pairs: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/missing_root_cell_pairs">3 <span class="ptr">evidence/coverage.json#/missing_root_cell_pairs</span></span></li>
<li>Excluded nested task dirs: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_dirs_count">0 <span class="ptr">evidence/coverage.json#/excluded_dirs_count</span></span></li>
<li>Excluded auxiliary top-level: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_auxiliary_data/top_level_path"><code>viewer_data</code> <span class="ptr">evidence/coverage.json#/excluded_auxiliary_data/top_level_path</span></span>; files: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_auxiliary_data/file_count">2397 <span class="ptr">evidence/coverage.json#/excluded_auxiliary_data/file_count</span></span>; directories: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_auxiliary_data/directory_count">1 <span class="ptr">evidence/coverage.json#/excluded_auxiliary_data/directory_count</span></span>; counted as task root: <span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/excluded_auxiliary_data/counted_as_task_root">false <span class="ptr">evidence/coverage.json#/excluded_auxiliary_data/counted_as_task_root</span></span></li>
</ul></div>

<h2>Coverage Matrix</h2>
<table>
<tr><th>Benchmark</th>
<th>Qwen3-1.7B-Base</th>
<th>Qwen3-4B-Base</th>
<th>SmolLM3-3B-Base</th>
<th>Gemma-3-4B-PT</th>
</tr>
<tr><td><code>aime2025</code></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/aime2025/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/aime2025/0</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/aime2025/1">48 <span class="ptr">evidence/coverage.json#/cell_counts/aime2025/1</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/aime2025/2">48 <span class="ptr">evidence/coverage.json#/cell_counts/aime2025/2</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/aime2025/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/aime2025/3</span></span></td>
</tr>
<tr><td><code>arenahardwriting</code></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/arenahardwriting/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/arenahardwriting/0</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/arenahardwriting/1">48 <span class="ptr">evidence/coverage.json#/cell_counts/arenahardwriting/1</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/arenahardwriting/2">50 <span class="ptr">evidence/coverage.json#/cell_counts/arenahardwriting/2</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/arenahardwriting/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/arenahardwriting/3</span></span></td>
</tr>
<tr><td><code>bfcl</code></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/bfcl/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/bfcl/0</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/bfcl/1">48 <span class="ptr">evidence/coverage.json#/cell_counts/bfcl/1</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/bfcl/2">48 <span class="ptr">evidence/coverage.json#/cell_counts/bfcl/2</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/bfcl/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/bfcl/3</span></span></td>
</tr>
<tr><td><code>gpqamain</code></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gpqamain/0">46 <span class="ptr">evidence/coverage.json#/cell_counts/gpqamain/0</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gpqamain/1">49 <span class="ptr">evidence/coverage.json#/cell_counts/gpqamain/1</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gpqamain/2">47 <span class="ptr">evidence/coverage.json#/cell_counts/gpqamain/2</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gpqamain/3">49 <span class="ptr">evidence/coverage.json#/cell_counts/gpqamain/3</span></span></td>
</tr>
<tr><td><code>gsm8k</code></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gsm8k/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/gsm8k/0</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gsm8k/1">47 <span class="ptr">evidence/coverage.json#/cell_counts/gsm8k/1</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gsm8k/2">49 <span class="ptr">evidence/coverage.json#/cell_counts/gsm8k/2</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/gsm8k/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/gsm8k/3</span></span></td>
</tr>
<tr><td><code>healthbench</code></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/healthbench/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/healthbench/0</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/healthbench/1">48 <span class="ptr">evidence/coverage.json#/cell_counts/healthbench/1</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/healthbench/2">48 <span class="ptr">evidence/coverage.json#/cell_counts/healthbench/2</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/healthbench/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/healthbench/3</span></span></td>
</tr>
<tr><td><code>humaneval</code></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/humaneval/0">47 <span class="ptr">evidence/coverage.json#/cell_counts/humaneval/0</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/humaneval/1">47 <span class="ptr">evidence/coverage.json#/cell_counts/humaneval/1</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/humaneval/2">48 <span class="ptr">evidence/coverage.json#/cell_counts/humaneval/2</span></span></td>
<td><span class="evidence-result" data-evidence-pointer="evidence/coverage.json#/cell_counts/humaneval/3">48 <span class="ptr">evidence/coverage.json#/cell_counts/humaneval/3</span></span></td>
</tr>
</table>

<h2>Reward-Hacking Statuses</h2>
<div class="card"><p>Contamination: <span class="evidence-result" data-evidence-pointer="evidence/reward_hacking.json#/training_on_test_sets/status"><span class="status status-partial">partial-support</span> <span class="ptr">evidence/reward_hacking.json#/training_on_test_sets/status</span></span> &middot; Instruction model: <span class="evidence-result" data-evidence-pointer="evidence/reward_hacking.json#/downloading_instruction_tuned_checkpoint/status"><span class="status status-partial">partial-support</span> <span class="ptr">evidence/reward_hacking.json#/downloading_instruction_tuned_checkpoint/status</span></span> &middot; API misuse: <span class="evidence-result" data-evidence-pointer="evidence/reward_hacking.json#/using_discovered_api_key/status"><span class="status status-unavailable">unavailable</span> <span class="ptr">evidence/reward_hacking.json#/using_discovered_api_key/status</span></span></p></div>

<h2>Limitations</h2>
<div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/limitations/0">No H100 run is reproduced; the resource and time findings are a released-configuration audit. <span class="ptr">evidence/claims.json#/claim_1/limitations/0</span></span></div>
<div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/limitations/1">The runner allows a five-minute termination grace, and a released example records 10:05:01. <span class="ptr">evidence/claims.json#/claim_1/limitations/1</span></span></div>
<div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/limitations/2">The pinned source's current launcher is scheduler-dependent: one branch has a 100-hour/eight-GPU METR command, another has 7 ten-hour and 1 one-hour default-GPU commands, and the arrays currently select only one model/benchmark pair. <span class="ptr">evidence/claims.json#/claim_1/limitations/2</span></span></div>
<div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_1/limitations/3">Evidence is not an official challenge verdict. <span class="ptr">evidence/claims.json#/claim_1/limitations/3</span></span></div>
<div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/limitations/0">A released judge label is not independently established behavioral truth. <span class="ptr">evidence/claims.json#/claim_2/limitations/0</span></span></div>
<div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/limitations/1">The instruction-model evidence is an upstream categorical label plus safe extracts from a released trace, not a fresh independent behavioral audit. <span class="ptr">evidence/claims.json#/claim_2/limitations/1</span></span></div>
<div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/limitations/2">The selected trajectory revision does not expose the exact GPT-5.1 Codex-Max API-misuse task cluster 16804408. <span class="ptr">evidence/claims.json#/claim_2/limitations/2</span></span></div>
<div class="limitation"><span class="evidence-result" data-evidence-pointer="evidence/claims.json#/claim_2/limitations/3">No leaderboard score, BFCL score, weighted average, or reasoning-effort ablation is a selected target or reproduced measurement. <span class="ptr">evidence/claims.json#/claim_2/limitations/3</span></span></div>

<footer>
<p>This evidence is not an official challenge verdict.
<span class="ptr">evidence/claims.json</span>
<a href="index.html">Full evidence summary</a> &middot;
<a href="report.html">Detailed report</a></p>
</footer>
</body>
</html>