| # Conclusion | |
| --- | |
| <!-- trackio-cell | |
| {"type": "markdown", "id": "cell_681c8a481ed5", "created_at": "2026-07-31T22:14:14+00:00", "title": "Overall findings"} | |
| --> | |
| **Overall finding.** The reproducible public artifacts support 2 of the 6 requested claims and falsify 4. The agent leaderboard direction is robust in the released aggregate: agents beat baseline code but remain below human experts, and parallelization/batching are favorable relative task types. The dataset-characteristic claims are the weakest: task counts and patch-size statistics do not reproduce from the pinned Hugging Face dataset. | |
| **Reproducibility notes.** | |
| | Item | Note | | |
| | --- | --- | | |
| | Primary dataset | `https://huggingface.co/datasets/formulacode/formulacode-all/tree/897e48cab8a27d32ba20ddb970b1fc397d96ee95` | | |
| | Official eval harness | `https://github.com/formula-code/fc-eval/tree/c08f665e7bf3b4de225b72dc02ce9b15b7aaba2b` | | |
| | Official site data | `https://github.com/formula-code/formula-code.github.io/tree/a79001bfdf46cd6c6e11e742f8b98893294cbc7a` | | |
| | Project site | `https://formula-code.github.io` | | |
| | Paper | `https://arxiv.org/abs/2603.16011` and `https://openreview.net/forum?id=WArbqRUsAe` | | |
| | HF Job | `https://huggingface.co/jobs/Srishti280992/6a6d1d066b79c09949c1dc23` | | |
| | Public Hub model reference | `https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct` | | |
| I did not claim a full paper replication: the original scrape, environment synthesis, statistical benchmark validation, and proprietary-agent inference were outside the scope of this local audit. The provided script is deterministic over the public release, downloads all pinned parquets, hashes source files, recomputes task/patch statistics, and summarizes first-party website result JSON. This makes the falsifications concrete but release-scoped: they show that several paper numbers are not reproducible from the artifacts now available to readers. | |