| # Claim 4: Optimization scale |
|
|
|
|
| --- |
| <!-- trackio-cell |
| {"type": "markdown", "id": "cell_dc9c70d6f280", "created_at": "2026-07-31T22:14:13+00:00", "title": "Verdict: falsified"} |
| --> |
| **Claim.** Agents perform better at function-level optimizations than repository-level ones, with OpenHands + Claude 4.0 Sonnet showing strongest module-level performance that degrades at function-level, while OpenHands + GPT-5 shows the inverse pattern. |
|
|
| **Verdict: falsified as a blanket statement.** The specific OpenHands model patterns are reproduced, but the released stratified aggregate contradicts "agents perform better at function-level optimizations." In `f2_stratified.json` from `https://github.com/formula-code/formula-code.github.io/tree/a79001bfdf46cd6c6e11e742f8b98893294cbc7a`, mean function-level Advantage is -0.032143, while mean module-level Advantage is -0.003323. Module-level is therefore less negative overall. |
|
|
| | Slice | Recomputed Advantage | |
| | --- | ---: | |
| | Mean function-level | -0.032143 | |
| | Mean class-level | -0.028500 | |
| | Mean module-level | -0.003323 | |
| | OpenHands + Claude function | -0.003835 | |
| | OpenHands + Claude module | 0.295726 | |
| | OpenHands + GPT-5 function | 0.048876 | |
| | OpenHands + GPT-5 module | -0.009541 | |
|
|
| So the nuanced model interaction is credible: OpenHands + Claude is strongest at module scale and OpenHands + GPT-5 is better at function than module. The broader statement is not supported by the released aggregate. The recomputation is in `outputs/formulacode_audit/summary.json` and was repeated by HF Job `https://huggingface.co/jobs/Srishti280992/6a6d1d066b79c09949c1dc23`. |
|
|
| Related artifacts: dataset `https://huggingface.co/datasets/formulacode/formulacode-all/tree/897e48cab8a27d32ba20ddb970b1fc397d96ee95`, eval repo `https://github.com/formula-code/fc-eval/tree/c08f665e7bf3b4de225b72dc02ce9b15b7aaba2b`, paper `https://arxiv.org/abs/2603.16011`. |
|
|
