Claim 4: Optimization scale
Claim. Agents perform better at function-level optimizations than repository-level ones, with OpenHands + Claude 4.0 Sonnet showing strongest module-level performance that degrades at function-level, while OpenHands + GPT-5 shows the inverse pattern.
Verdict: falsified as a blanket statement. The specific OpenHands model patterns are reproduced, but the released stratified aggregate contradicts "agents perform better at function-level optimizations." In f2_stratified.json from https://github.com/formula-code/formula-code.github.io/tree/a79001bfdf46cd6c6e11e742f8b98893294cbc7a, mean function-level Advantage is -0.032143, while mean module-level Advantage is -0.003323. Module-level is therefore less negative overall.
| Slice | Recomputed Advantage |
|---|---|
| Mean function-level | -0.032143 |
| Mean class-level | -0.028500 |
| Mean module-level | -0.003323 |
| OpenHands + Claude function | -0.003835 |
| OpenHands + Claude module | 0.295726 |
| OpenHands + GPT-5 function | 0.048876 |
| OpenHands + GPT-5 module | -0.009541 |
So the nuanced model interaction is credible: OpenHands + Claude is strongest at module scale and OpenHands + GPT-5 is better at function than module. The broader statement is not supported by the released aggregate. The recomputation is in outputs/formulacode_audit/summary.json and was repeated by HF Job https://huggingface.co/jobs/Srishti280992/6a6d1d066b79c09949c1dc23.
Related artifacts: dataset https://huggingface.co/datasets/formulacode/formulacode-all/tree/897e48cab8a27d32ba20ddb970b1fc397d96ee95, eval repo https://github.com/formula-code/fc-eval/tree/c08f665e7bf3b4de225b72dc02ce9b15b7aaba2b, paper https://arxiv.org/abs/2603.16011.