Reproducing 147 Machine Learning Papers in 16 Days on One Workstation

A systems retrospective on the ICML 2026 Reproducibility Challenge

Nate Mauer, August 2026

---

Abstract

Between July 18 and August 2, 2026, I competed solo in the ICML 2026 agent reproduction challenge, a public competition hosted by Hugging Face and alphaXiv in which entrants reproduce the claims of ICML papers and an LLM judge scores the evidence claim by claim. I entered three days after the window opened. By the close, the judge had scored 147 of my reproductions for 1,390 points and 76 perfect scores, placing me 9th of 372 entrants on the scored leaderboard (roughly 1,200 participants registered). After a hand-run first wave that cost about $52, I imposed a $20 ceiling on all subsequent paid inference, and the automated campaign's metered total was about $2.69. All reproduction work ran on a single workstation and its one 20 GB GPU; the sole exception was one rented-GPU fine-tuning job.

The result was not produced by hand-crafting 147 reproductions. It was produced by treating the competition as a systems problem: a crash-safe 24x7 orchestration fleet, a fail-closed intake pipeline that ranked a 6,341-paper catalog by expected points per bottleneck-hour, three zero-cost verification gates that ran before any paid model call, a locally served 12B judge-distillation model that eliminated roughly 90 percent of paid judging calls, and a publish-early, upgrade-forever strategy built around the platform's rate limits. This report summarizes the architecture, the empirical findings about the judge and the reproducibility landscape, the results, and what the project demonstrates about my engineering practice. Every externally checkable number cited here can be audited against the competition's public record: the official leaderboard, the public verdicts dataset that records every verdict with its rationale, and the reproduction Spaces published under my public ParetoOptimal Hugging Face account, each of which contains the full evidence logbook and a deterministic entrypoint. The remaining figures come from the campaign's internal ledgers and receipts.

1. The competition

The challenge asked a simple question at an uncomfortable scale: how much recently published machine learning research actually holds up when someone tries to run it? Entrants picked papers from a catalog of 6,341 ICML submissions, attempted to reproduce each paper's officially anchored claims, and published the evidence as a static Hugging Face Space containing a Trackio logbook. A judge Space scanned for tagged submissions every minute and, every five minutes, asked the GLM-5.2 model for a verdict on each claim.

Scoring was per claim: a verified claim earned 2 points, a falsified claim also earned 2 points, an honest reduced-scale ("toy") reproduction earned 1, and an inconclusive attempt earned 0. Papers carried three to six anchored claims, so a paper was worth 6 to 12 points at maximum. Two properties of this rubric shaped my entire strategy. First, falsification paid exactly as well as verification, which means honesty about a paper's failures is not a consolation prize but a full-credit outcome. Second, the judge graded only the official claim list and treated every self-reported verdict as an untrusted author assertion, so the only thing that mattered was whether the judge could see commands, outputs, and numbers inside its reading window.

2. Strategy: selection is the highest-leverage problem

With a 16-day window and one machine, choosing which papers to attempt mattered more than any other decision. I built the selection layer as a data pipeline rather than a judgment call.

A miner swept all 6,341 catalog abstracts and found 1,128 papers linking a code repository. Static scoring ranked them, and the top candidates were live-probed with git ls-remote, a constant-time check with no meaningful rate limit; 145 of the top 150 repositories were alive. A release preflight then killed vaporware before any effort was spent: a repository whose contents listing held fewer than three substantive entries beyond a README and license was classified as a shell, and roughly half the greenfield queue failed that test. A second tier cross-referenced the competition's public verdicts dataset to find papers where another team had already scored well, using their success strictly as a tractability signal and never as evidence.

Two empirical regularities, learned from my own score data, were encoded directly into the ranker. Papers with runnable official code could reach fully verified outcomes, while pure-theory papers capped near half credit because the judge treats a simulation of a small instance as evidence for the instance, not the general theorem. And claim style was predictive enough to regex: executable claims (benchmarks, accuracies, speedups) were routed forward while theory claims carried a hard 0.5 expected-capture ceiling. On top of this sat a Thompson-sampling bandit over release-quality and compute-class arms, with priors seeded from measured campaign outcomes, and a deterministic ordering by expected points per bottleneck-hour with a 10 percent exploration quota drawn only from the lower-yield half of the eligible pool.

The operating doctrine, written into the fleet's operating contract, was to track points per bottleneck-hour, not paper count.

3. Architecture: the fleet

The system that executed this strategy, the fleet, ran in two generations.

The first generation was a restart-safe 24x7 supervisor installed as a Windows scheduled task with an infinite-restart wrapper. It owned a single crash-safe JSON state store guarded by advisory file locks, so a second supervisor instance would refuse to start rather than corrupt the ledger. Each pass ran a bounded, read-only intake inspector that wrote one immutable receipt per paper keyed by manifest and paper SHA-256, sorted papers into four lanes (perfect-score, normal-throughput, release-audit-only, and quarantined), and demoted any paper whose claims were not all directly testable or whose expected capture fell below 90 percent. The per-paper orchestrator then drove LLM coding-agent workers, initially the OpenAI Codex CLI and later a second agent harness for the endgame batches, through reproduction, gating, screening, and a single metered judge call, with a two-hour worker timeout and at most two constrained repair cycles per failure class.

The second generation, built for the final week's push, decomposed the same pipeline into small single-purpose drivers for gating, the paid final, publishing, and resync, each emitting one JSON line and each refusing to act without the receipt produced by the stage before it. A cap-aware scheduler decided every batch: it tracked Hugging Face's Space-creation rate limit through a creation ledger, treated an observed HTTP 429 as authoritative over its own estimate, and split work between new papers, which consume a scarce creation slot, and headroom papers, which are already published and can be upgraded in place without limit.

The pipeline, end to end: select, reproduce, gate, screen, pay once, publish, watch, learn. A verdict watcher polled the official dataset with conditional HTTP requests and fed every new verdict back into the bandit, and a harvest step archived each trajectory into a content-addressed store, 3,442 artifacts across 146 trajectories, for later training use.

4. Three free gates before any dollar

The cost discipline of the whole system rests on an ordering principle: each verification stage is strictly cheaper than the stage it protects.

Gate one is determinism. Every workspace exposes a validate.py entrypoint that is run twice and byte-compared on stdout and stderr, which catches unseeded randomness, wall-clock leakage, and dictionary-ordering drift before anything else sees the work.

Gate two is a byte-faithful simulation of the judge's reading behavior, the judge-window simulator. The judge reads logbook pages alphabetically after the index under a 120,000-character budget, strips HTML fences, and truncates raw blocks to 4,000 characters. The simulator replicates that ingestion exactly and hard-fails unless every page is fully visible and every must-see evidence string survives the window.

Gate three is the integrity gate. Any numeric token appearing in claim-bearing prose must exist verbatim in a raw results file, and a secret scan runs over everything publishable. This rule produced one of the project's better patterns: pages are generated by a small script that interpolates numbers directly from the raw results JSON, so every published figure is a verbatim substring of its source by construction.

Only after all three gates does a paper reach the free local screener, then exactly one paid GLM-5.2 call per unique logbook content digest, cached forever, and only then a publish slot. Wave 1, run by hand before this machinery existed, cost about $52 for 8 papers. The full campaign that followed cost about $2.69 for everything else.

5. The $0 judge: distilling the grader

The single most consequential component was a local model trained to predict the judge. I reconstructed the judge's system prompt and ingestion byte-identically from its public Space, verified the reconstruction against known verdicts, and built a training corpus of 1,850 reconstructed judge views from the competition's public verdicts dataset, split by paper so no paper crossed the train and held-out boundary.

The student is Gemma 4 12B fine-tuned with QLoRA. The training story is honest about its failures: nine documented local attempts on my 20 GB GPU failed in instructive ways, including a WSL2 memory blowup that took down the entire VM and a multimodal-processor quirk that silently routed text into the image pathway. The successful run took 74 minutes on a rented two-H200 job. The result was merged, quantized to a 7.38 GB GGUF with pinned checksums, and served locally by llama.cpp with a 131,072-token context split into two slots, which matters because judge views routinely run past 50,000 tokens.

Evaluated against the real judge on 74 held-out papers and 201 claims, the student reached 70.15 percent claim agreement, an F1 of 0.83 on verified claims, a micro-F1 of 0.70, and a 6.7 percent optimistic bias. My own promotion gate required 85 percent agreement before the student could grade anything on its own authority. It failed that gate, so the manifest hard-codes it as advisory only.

A failed grader still turned out to be the most valuable model in the system, because its one reliable property is exploitable: it flatters. A model with an optimistic bias that still predicts a submission will fall short is giving testimony against its own tendency, and that negative signal is high-trust. Routing on exactly that signal, the fleet skipped the paid judge call for any paper the student did not score as perfect, which cut paid judging calls by roughly 90 percent while never letting the student approve anything.

The same discipline governed its retirement. A v3 retrain was fully staged, with the exact rented-GPU command written into a plan file, and then declined at zero cost because the estimate of $47 to $61 exceeded the campaign's $20 training ceiling and the held-out label support had not met its per-class floor. That decision was itself recorded in a receipt alongside the staged plan.

6. Measuring the grader

The competition's hidden curriculum was that the judge itself is a system you must characterize empirically, and my most expensive lessons became the system's most valuable rules.

My first submission scored 0 of 4 because a 1.37-million-character page pushed the actual evidence out of the judge's 120,000-character window. That failure became the judge-window simulator. An early paper scored 1 of 4 because I had verified claims the judge was not grading; the grader scores only the officially anchored claim list, and when the organizers switched from a coarse three-claim file to a granular anchored set mid-competition, a hand-made claims file that predicted 6 of 6 scored 0 of 12 against the real rubric. That burn became a fail-closed check that blocks any paper whose local claims snapshot mismatches the official one, before any money is spent.

Reading the judge's own rationales at scale produced a working definition of its "toy" verdict: a single small instance, no parameter sweep, no realistic scale, a missing baseline the paper compares against, or synthetic data where the paper used real data. The corresponding fix, scale plus sweep plus coverage with named baselines and multiple seeds, was written into every upgrade prompt. The judge also exhibits verdict variance, and I observed a verified claim flip to inconclusive on identical evidence after a re-sync, so a satisfactory verdict triggers an immediate freeze of the Space.

The deepest finding is about the rubric's incentive design: because falsification pays full credit and hedging pays nothing, the rational strategy and the honest strategy coincide. The papers where I proved something wrong scored as well as the papers where I confirmed everything.

7. Operating around platform limits

Hugging Face caps Space creation at 20 per rolling 24 hours per account, a rate limit that a paid plan does not lift. Updating an existing Space, however, is uncapped, and the judge re-scores on every change. That asymmetry became the endgame strategy: a Space is a slot you buy once, and evidence is a payload you can upgrade forever.

So the fleet published early to claim slots, then improved in place. The resync driver updates a live Space through the commit API rather than the rate-limited creation endpoint, deletes stale remote pages explicitly, re-downloads every page afterward, and byte-compares the remote against the local gate digest before re-activating tags. Papers are matched to Spaces by their competition tag rather than their slug, because a handful of historic slugs were nonstandard and a slug-based republish would have created a duplicate and burned a creation slot.

The receipts show what this was worth. One night of in-place upgrades moved five already-published papers from 8 combined official points to 19. On July 28 I launched a targeted lift of the 16 published papers scoring under 50 percent, feeding each worker the judge's verbatim rationale for every lost point, and continued with six more rounds the following day. One paper regressed from 3 to 0 when a rewrite mismatched the judge's claim set, and then recovered, which is the risk the no-downgrade guard was subsequently built to prevent: a republish is allowed only when the local screener predicts at least the current official score.

8. Results

The trajectory over the window, from standing snapshots taken from the public leaderboard during the campaign:

The final figures: 147 papers judged, 1,390 points, 76 perfect scores, about $2.69 of paid inference, 9th place among 372 scored entrants out of roughly 1,200 registered participants, after entering three days late.

Per-wave capture, from the local ledger (which mirrors verdicts through July 28; the final week's scores live on the public leaderboard; the earliest waves predate the ledger, and the waves judged after the mirror cutoff appear only on the leaderboard):

Calibration held up: across the 39 papers with both a paid pre-judgment and an official verdict, the system predicted 232 points and the judge awarded 235.

A few reproductions are worth naming individually.

Provably Data-driven Lagrangian Relaxation for MILP scored a perfect 12 of 12 with two of its six claims falsified by exact counterexamples: the paper's own sufficient condition degenerates to an impossible bound at its advertised parameter value, and a lower-bound construction in the appendix uses a probability vector that sums to 1.2. The entire analysis ran on CPU for zero dollars, and both errors are checkable by hand.

Finite-time Convergence Analysis of Actor-Critic with Evolving Reward is a pure-theory paper with no released code. I implemented its algorithm from scratch in NumPy, measured the paper's own convergence quantities on a 50-state MDP, recovered the predicted rate (a log-log slope of -0.533 with an R-squared of 0.973), and in the process found four internal contradictions between the paper's theorem statements and its appendix. It scored 10 of 10 at high quality, from roughly seven minutes of CPU time.

Budget-Feasible Mechanisms for Submodular Welfare Maximization shows the recovery loop. It first scored 1 of 12. The rebuild compiled the authors' unmodified C++ and ran it on the three real SNAP graphs that the paper's node counts uniquely identify, reproducing the claimed improvement range to four decimal places and matching a digitized figure to a median relative error of 0.0009; the original failure traced to substituting a wrong graph. After a second repair pass the local prediction reached 12 of 12.

The failures are part of the record too. One paper scored 0 of 10 with every claim inconclusive, an agent-memory paper scored 2 of 8 on its first pass, and the vaporware archetype, a paper whose release table listed every artifact as "in preparation," scored 1 of 4 and seeded the bandit's priors for that class.

9. Integrity as an architectural property

A competition scored by an LLM invites gaming, so I made honesty a property the system enforces rather than a resolution I hoped to keep. The optimization loop's action space is restricted to three moves: making real evidence more visible, adding real evidence from new experiments, and sharpening honest verdicts, for example refusing to hedge a genuine falsification into a zero-point inconclusive. Rewording without new evidence is not in the action space, and the integrity gate makes fabrication mechanically difficult because prose numbers must exist in raw output files.

The clearest test of this posture was the partial-credit lane. My publisher initially required a predicted perfect score, which quarantined dozens of honest 8-of-12 reproductions that were worth real points. The fix was a deliberately narrow operator tool that drops exactly one check, the perfect-credit comparison, while retaining every digest, gate, claim-count, and evidence-visibility verification; it dry-runs by default, requires a written reason to apply, and writes a two-stage receipt for every release. About 43 published papers flowed through that lane, and every one of them left its receipts in the campaign's records.

The worker playbooks state the doctrine outright: a genuine 3 of 6 is a success, and a fabricated 6 of 6 is misconduct.

10. Lessons learned

Constraints compose, so solutions must too. A budget ceiling, a creation rate limit, a judge reading window, one GPU, and a 16-day clock do not add up; they multiply. No single clever trick survives all of them. The only thing that did was a pipeline in which every stage is cheaper than the stage it protects and every decision leaves a receipt.

Measure the grader, not just the work. Perfect reproductions scored zero when the evidence fell outside the judge's window or outside its claim list. Characterizing the evaluator, byte-for-byte where possible, was worth more points per hour than improving any individual reproduction.

Throughput lives in the infrastructure. The difference between 8 papers in the hand-run first wave and 147 judged by the close was not working harder; it was a supervisor that survived crashes, receipts that made every step idempotent, and a scheduler that respected the platform's limits instead of fighting them.

A model that fails its promotion gate can still be the most valuable model in the system, provided you measure its failure precisely and route only on the signals it gets right.

Honesty was the dominant strategy, not a constraint on one. Falsifications scored full credit, hedges scored nothing, and the integrity gates meant the cheapest path to points was real evidence presented clearly.

11. Conclusion: what this demonstrates

I built this system as a graduate student finishing an MS in Computer Science, working alone, on hardware I already owned apart from a single rented GPU, against roughly 1,200 registered entrants that included teams with far larger budgets. I finished 9th on the scored leaderboard of 372, with more paid inference spent on a typical single API demo than I spent on the entire campaign.

For a hiring manager, I think the transferable claims are these. I can design and operate agentic LLM pipelines at scale: hundreds of autonomous reproduction jobs, each gated, budgeted, and auditable, running unattended for two weeks. I can do evaluation engineering, which I would argue is the scarcest skill in applied ML right now: reconstructing a grader byte-for-byte, distilling it into a local model, measuring that model honestly enough to demote it, and still extracting most of its value. I can do cost engineering with real teeth, holding a hard $20 ceiling through digest-cached inference, prefiltering, and receipts, and declining a prepared $50 training run because the numbers said no. I can build crash-safe, idempotent distributed systems on unglamorous infrastructure, a Windows workstation with WSL, file locks, and scheduled tasks, where correctness came from single-owner state and immutable receipts rather than from a cluster. And I can hold scientific standards under adversarial incentives, because the system I built makes fabrication harder than honesty and the record shows the low scores next to the perfect ones.

The competition asked how much published machine learning research is reproducible. My data says the answer is uneven: papers that ship runnable code mostly verify, theory papers mostly cannot be tested past their small instances, about half of the greenfield queue was vaporware, and a nontrivial fraction of published proofs fail on arithmetic you can check by hand. Producing that answer at a rate of nine papers a day is what I now know how to do, and the receipts are public: every reproduction Space is live under my ParetoOptimal Hugging Face account, and every verdict, rationale included, sits in the competition's public verdicts dataset for anyone to audit.

Appendix: data provenance

Numbers in this report come from two kinds of sources, the competition's public record and the campaign's internal ledgers and receipts, and a careful reader will find three places where figures could seem to disagree. First, spend: the public figure of about $2.69 is the metered total from router-reported costs; the internal ledger reads $7.19 because the late-campaign driver conservatively charges a $0.05 fallback whenever the router omits a cost estimate, and a per-call summation of recorded costs gives $2.51. The metered figure is the accurate bill; the ledger is deliberately pessimistic because it guards a hard ceiling. The hand-run first wave's roughly $52 predates both the ceiling and the metering, sits outside all three of those figures, and is recorded separately in the internal ledgers. Second, coverage: the local state ledger mirrors official verdicts only through July 28 (63 papers, 382 of 624 points), because the final week's verdicts were read from the public leaderboard directly; the closing totals of 147 papers and 1,390 points are the official judged figures recorded on August 2. Third, the first wave's tally appears as both 23 points across 8 papers and 20 across 7 in the campaign's internal summaries, the difference being one paper's inclusion; and the leaderboard denominator grew from 202 to 372 over the window as entrants earned their first verdicts, with total registrations near 1,200. What a reader can verify publicly: the closing leaderboard standing, every per-claim verdict and its rationale in the competition's public verdicts dataset, and every published reproduction Space under my ParetoOptimal Hugging Face account, each carrying its full evidence logbook and a deterministic entrypoint. What comes from internal bookkeeping: the spend ledgers, the gate and release receipts, and the prediction calibration records.

Links: my published reproduction Spaces live under huggingface.co/ParetoOptimal, every verdict and rationale sits in the public verdicts dataset (ICML-2026-agent-repro/verdicts), and the official leaderboard is on the competition Space.