Picking reproducible papers out of a 6,341-paper haystack
The reproduction challenge offers 6,341 candidate papers and 18 days. Working one paper takes hours to days, so at most a few hundred attempts fit in the window. Selection is worth more than speed: reproducing the wrong paper costs a day and returns nothing, while the right paper returns full points for an afternoon of work. I treated selection as its own engineering problem, and the tooling paid for itself within the first sweep.
What actually predicts points
After a few dozen shipped reproductions, my score data sorted papers into a clear hierarchy:
Papers with runnable official code reach verified. When the authors ship code and the data is downloadable, the judge sees the authors' own program producing the claimed numbers, and full-credit verdicts follow. My best results all fit this pattern: driving an official mechanism-design release across a parameter sweep, compiling the authors' C++ and reading its oracle counts, running the authors' uncertainty-quantification code on the real UCI datasets it named.
Pure-theory papers cap at partial. I reproduced theorem after theorem with clean simulations, and the judge consistently ruled them partial: a simulation of a 1D instance is evidence for the instance, not the general theorem. Fitting a scaling law with an R^2 of 0.999 does not change that ceiling. Expected capture for theory sits near half credit, and no amount of simulation polish moves it.
Someone else's success is the strongest signal of all. The challenge publishes verdicts for everyone, which means every other team's result is free reconnaissance. If another competitor scored full credit on a paper, that paper is proven reproducible under the real judge, whatever my priors say about its type.
The miner
The full catalog mirror lives locally, so the search is a data pipeline rather than a browsing session:
Tier A: leader-proven. Cross-reference the public verdict data with my attempt history. The sweep found 33 unattempted papers where another team already scored, most at full credit. These carry an observed capture fraction instead of a guess, which makes them the highest-confidence targets in the catalog.
Tier B: code-bearing. A regex over all 6,341 abstracts finds real repository links: 1,128 papers link GitHub, GitLab, or Hugging Face repos. One trap worth naming: the catalog's hf field looks like a release signal but holds an arXiv id, a paper page rather than code. Trust URLs you extracted yourself.
Probe before you trust. Abstract links rot. The cheap test is git ls-remote, which confirms a repository exists and returns its HEAD for one unauthenticated request, effectively O(1) per repo with no meaningful rate limit. Of my top 150 candidates, 145 were alive. The 5 dead links would each have wasted an evening.
Audit before you commit. The top survivors get a rate-budgeted GitHub API pass: repo size, license, last push, whether an environment file and runnable scripts exist. This caught my favorite trap of the campaign, a paper whose linked repo was a 1 KB empty shell. The abstract promised code. The repository contained a README and nothing else.
Score what remains. Expected points = the paper's maximum times an observed capture rate for its type, taken from my own shipped results rather than optimism. Divide by estimated hours, rank, and exclude the 257 papers already attempted. Ranking by expected points per hour is really picking the Pareto frontier of the catalog: the papers you cannot beat on points without spending more time. Claims themselves get a keyword split into executable versus theorem-shaped, because a paper whose claims name datasets and accuracy numbers is a different job from one whose claims name propositions, and on points per hour, the code-bearing job dominates the theorem-shaped one whatever the abstracts promise.
What transfers
- Selection compounds. An hour of ranking beats a day of reproducing the wrong paper, every time the catalog is large.
- Build the hierarchy from your own outcome data, not from intuition about what looks easy. My intuition said theory papers were quick wins. My score data said they were capped.
- Other people's public results are reconnaissance. Proven-reproducible dominates predicted-reproducible.
- Verify artifacts exist before scheduling work against them. The cheapest probe available, one
git ls-remoteper candidate, filtered out every dead end for pennies of compute. - Empty repositories are a real failure mode. A link is a claim, not a release.