Conclusion
Overall finding. The reproducible public artifacts support 2 of the 6 requested claims and falsify 4. The agent leaderboard direction is robust in the released aggregate: agents beat baseline code but remain below human experts, and parallelization/batching are favorable relative task types. The dataset-characteristic claims are the weakest: task counts and patch-size statistics do not reproduce from the pinned Hugging Face dataset.
Reproducibility notes.
| Item | Note |
|---|---|
| Primary dataset | https://huggingface.co/datasets/formulacode/formulacode-all/tree/897e48cab8a27d32ba20ddb970b1fc397d96ee95 |
| Official eval harness | https://github.com/formula-code/fc-eval/tree/c08f665e7bf3b4de225b72dc02ce9b15b7aaba2b |
| Official site data | https://github.com/formula-code/formula-code.github.io/tree/a79001bfdf46cd6c6e11e742f8b98893294cbc7a |
| Project site | https://formula-code.github.io |
| Paper | https://arxiv.org/abs/2603.16011 and https://openreview.net/forum?id=WArbqRUsAe |
| HF Job | https://huggingface.co/jobs/Srishti280992/6a6d1d066b79c09949c1dc23 |
| Public Hub model reference | https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct |
I did not claim a full paper replication: the original scrape, environment synthesis, statistical benchmark validation, and proprietary-agent inference were outside the scope of this local audit. The provided script is deterministic over the public release, downloads all pinned parquets, hashes source files, recomputes task/patch statistics, and summarizes first-party website result JSON. This makes the falsifications concrete but release-scoped: they show that several paper numbers are not reproducible from the artifacts now available to readers.