Really enjoyed being part of this experiment. Gratifying to see our reproduction highlighted as one of the community’s best.
“A Coin Flip for Safety” was reproduced using AlphaXiv’s OpenResearch with OpenCode’s free BigPickle model, with experiments run locally on 2× NVIDIA RTX 3090s.
It was a great demonstration of how agent-assisted research can make independent reproduction and auditing accessible with relatively modest compute.
Thanks to the Hugging Face and AlphaXiv teams for organizing the challenge and making the whole process open and reproducible!
One other difference: every attempt runs under roar (python package roar-cli), which passively captures provenance rather than relying on code instrumentation like Trackio or W&B. That gives us bilibili the actual lineage of code, data, environment and artifacts, automatically producing an AI-BOM and a roar reproduce path others can independently repeat.
It was interesting to reproduce one of the ICML accepted papers using these tools.
This analysis from the hackathon was particularly insightful, and agent-based reproducibility has the potential to become a de facto standard for strengthening the credibility of claims in ML conferences. It could also serve as a valuable pathway toward refining autonomous scientific discovery, revealing not only where agents succeed, but also where they fail, providing important insights into their limitations and how human-agent collaboration can be improved.
The scale of this experiment is honestly incredible 🤯... Reproducing 2,226 ICML papers in just 19 days shows how quickly coding agents are changing the research workflow. What once took researchers days or even weeks can now be tested in parallel, giving us a much larger picture of what actually holds up when the claims are checked carefully. 🔬💻
What I find especially interesting is the human side of this shift... Agents can read papers, write code, run experiments, and generate reports, but deciding what the results actually mean still requires strong human judgment. A successful reproduction is not automatically proof that every claim is correct, and a failed reproduction can have many possible explanations. 🤔📊
The point about reviewing capacity is particularly important... When thousands of papers are being submitted and reviewers simply cannot examine every proof, experiment, or implementation in depth, scalable reproduction could become an important second layer of scientific scrutiny. It could help researchers focus their limited time on the most meaningful questions instead of spending every hour on repetitive experimental work. 🔎⚙️
Most importantly, this experiment suggests that AI agents may not replace researchers so much as change where researchers add the most value... Humans may increasingly become the people who design better questions, evaluate evidence, investigate unexpected results, and decide which findings deserve attention. That combination of human curiosity + agent-powered experimentation could make scientific research far more rigorous and scalable. 🚀🧠✨
