One workstation, 6,341 papers, 18 days: the workflow is the product
The ICML 2026 agent-reproduction challenge, run by Hugging Face and alphaXiv is from July 15 to August 2, 2026. The task sounds simple: pick published ML papers, reproduce or refute their claims with real executed experiments, and publish the evidence as a public Trackio logbook on the Hub. An official judge built on GLM-5.2 scores every anchored claim: 2 points for verified or falsified, 1 for a partial reproduction, 0 for inconclusive.
What makes it interesting is the constraint stack. The catalog holds 6,341 candidate papers. The window is 18 days. I gave the campaign a 20 dollar ceiling for paid inference. The Hub allows 20 new Spaces per rolling 24 hours. The judge reads at most 120,000 characters of each logbook. And I am one graduate student with one workstation and a WSL sidecar, not a lab.
Early on I noticed that every one of those constraints attacks a different part of the system. Budget attacks judging. The Space cap attacks shipping. The catalog size attacks selection. The context window attacks how evidence is written down. No single clever trick fixes all of them, so the real work of the competition became designing a workflow where each stage absorbs its own constraint.
Put another way, those constraints are competing objectives, namely points, dollars, publish slots, and hours, and no single move maximizes all of them at once. The goal is not to win any one axis but to sit on the Pareto frontier: the set of designs where you cannot earn another point without giving up dollars, slots, or time you do not have. The handle I ship these logbooks under, ParetoOptimal, is that thesis in one word, and every post in this series is really about holding that frontier for one stage of the pipeline.
This post is the map of that workflow. Four companion posts go deep on the individual pieces.
The pipeline, end to end
1. Mine the catalog. The full paper index lives locally, so selection is a data problem, not a browsing problem. A miner scores all 6,341 papers for expected points per hour of work: does the abstract link real code, are the claims executable or purely theoretical, has another team already proven the paper reproducible. The latest sweep surfaced 33 unattempted papers where another competitor already earned points, plus 145 live code-bearing repos ranked by feasibility. Details in picking reproducible papers.
2. Queue in waves. Selected papers become wave manifests processed by a crash-safe supervisor with a single state ledger and budget reservations. An intake audit sorts every paper into one of four lanes before any compute is spent, and a Thompson bandit keeps score of which paper types actually pay out, so selection improves as evidence accumulates. Details in waves-queues-and-gates.md.
3. Run genuine experiments. Each paper gets a workspace with a pinned manuscript, its claims, and a validator that runs real code: the authors' release when one exists, a faithful from-scratch implementation when it does not. The one rule with no exceptions: every number that appears on a logbook page must come from a raw result file produced by an actual run. Honest gaps are labeled as gaps.
4. Gate locally. Before anything ships, three free gates run: the validator must produce byte-identical output across two runs, a judge-window simulator confirms the evidence fits and is visible inside the official 120,000-character ingestion, and an integrity scanner rejects any page number that lacks raw-file support. The gates cost nothing and catch almost everything.
5. Screen with a $0 judge, then pay once. A local Gemma-4-12B model, fine-tuned on the challenge's public verdicts, predicts the official score of every gated bundle for free. Only promising candidates trigger the paid GLM-5.2 pre-judge, and each unique content digest pays at most once. That filter cut paid judging by roughly 90 percent: about 2.69 dollars of paid inference across the whole campaign. Details in the distilled judge.
6. Publish through two lanes. An automatic lane ships predicted-perfect papers. An operator lane ships everything above a quality bar with caveats stated plainly, because the rubric pays partial credit and a queue full of unshipped 8-of-12 bundles is a pile of unclaimed points.
7. Improve in place. The Space-creation cap meters new papers at 20 per day, but updates to an existing Space are uncapped, and the judge re-scores whenever a Space changes. So the workflow publishes early and then upgrades the evidence in place. One night of in-place reworks lifted five already-published papers from 8 combined official points to 19, with about 57 more points queued behind the cap. Details in [publish, then improve] Coming Soon.
What the workflow taught me
Constraints compose, so solutions must too. The distilled judge is only affordable because the gates are free. Partial-credit publishing only works because in-place updates are uncapped. The miner only matters because the queue can actually drain what it finds. Each piece was designed against one constraint, but the value comes from the joints. Composed well, they trace an efficient frontier: each stage pushes work onto the cheapest axis that can still do it, so the pipeline as a whole stays near Pareto-optimal even though no single stage is trying to be.
Measure the grader, not just the work. The single most expensive mistake available in this challenge is optimizing something the judge never reads. I simulate the judge's exact ingestion before shipping, and I learned the hard way that a prediction made against the wrong claim set can say 6 of 6 while the real rubric says 0 of 12. Eval quality means rubric match first and model quality second.
Throughput lives in the infrastructure. My coursework gave me the asymptotics, and they showed up everywhere: O(1) repo probes instead of rate-limited API calls, a screener that is O(1) per candidate in dollars, bounded judge windows, points-per-hour ranking. But the leaderboard rewards the plumbing that turns those into shipped papers. A schedule that respects the caps beats a sprint that discovers them, and that is the whole reason a late entry on one workstation now sits 11th of 352, on 1,066 points across 127 judged papers and under three dollars of paid inference.