# Claim 4: Optimization scale


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_dc9c70d6f280", "created_at": "2026-07-31T22:14:13+00:00", "title": "Verdict: falsified"}
-->
**Claim.** Agents perform better at function-level optimizations than repository-level ones, with OpenHands + Claude 4.0 Sonnet showing strongest module-level performance that degrades at function-level, while OpenHands + GPT-5 shows the inverse pattern.

**Verdict: falsified as a blanket statement.** The specific OpenHands model patterns are reproduced, but the released stratified aggregate contradicts "agents perform better at function-level optimizations." In `f2_stratified.json` from `https://github.com/formula-code/formula-code.github.io/tree/a79001bfdf46cd6c6e11e742f8b98893294cbc7a`, mean function-level Advantage is -0.032143, while mean module-level Advantage is -0.003323. Module-level is therefore less negative overall.

| Slice | Recomputed Advantage |
| --- | ---: |
| Mean function-level | -0.032143 |
| Mean class-level | -0.028500 |
| Mean module-level | -0.003323 |
| OpenHands + Claude function | -0.003835 |
| OpenHands + Claude module | 0.295726 |
| OpenHands + GPT-5 function | 0.048876 |
| OpenHands + GPT-5 module | -0.009541 |

So the nuanced model interaction is credible: OpenHands + Claude is strongest at module scale and OpenHands + GPT-5 is better at function than module. The broader statement is not supported by the released aggregate. The recomputation is in `outputs/formulacode_audit/summary.json` and was repeated by HF Job `https://huggingface.co/jobs/Srishti280992/6a6d1d066b79c09949c1dc23`.

Related artifacts: dataset `https://huggingface.co/datasets/formulacode/formulacode-all/tree/897e48cab8a27d32ba20ddb970b1fc397d96ee95`, eval repo `https://github.com/formula-code/fc-eval/tree/c08f665e7bf3b4de225b72dc02ce9b15b7aaba2b`, paper `https://arxiv.org/abs/2603.16011`.