# Executive summary
---
## Executive summary
This canonical logbook presents the public full-score reproduction evidence
from [Srishti280992/repro-formulacode-evaluating-agentic-optimization-on-large-codebases](https://huggingface.co/spaces/Srishti280992/repro-formulacode-evaluating-agentic-optimization-on-large-codebases) with explicit
attribution. Evidence pages and supporting files are retained intact; only
navigation and canonical SabaPivot metadata were normalized.
---
---
**Pinned summary.** I reproduced the FormulaCode claims against the paper, the pinned public dataset, and the first-party website/evaluation artifacts. The public release verifies the main agent-result direction and the optimization-type aggregate, but falsifies the released task-count, patch-size, blanket optimization-scale, and repository-popularity details. The audit was run locally and repeated as a Hugging Face Job; no GPU was used because the verification is data parsing, hashing, and arithmetic over released artifacts rather than model inference.
| Claim | Verdict | Main reason |
| --- | --- | --- |
| 1 | falsified | Pinned `formulacode/formulacode-all` has 1,215 default tasks, not 957. |
| 2 | falsified | Recomputed patches average 143.84 changed lines and 4.03 files; workload counts are absent from the task parquet. |
| 3 | verified | All 8 non-human agent rows are above 1.0x speedup and have negative Advantage; human global speedup is 1.10396x. |
| 4 | falsified | Specific OpenHands model patterns hold, but function-level mean Advantage is worse than module-level. |
| 5 | falsified | Non-monotonicity holds, but Q2 alone is strongest and Q4 alone is the low point. |
| 6 | verified | Parallelization and batching are positive; the lower-level/native/vectorized aggregate is below -0.2. |
**Scope and cost.**
| Item | This reproduction | Full replication |
| --- | --- | --- |
| Source artifacts | Paper `https://arxiv.org/abs/2603.16011`, OpenReview `https://openreview.net/forum?id=WArbqRUsAe`, dataset `https://huggingface.co/datasets/formulacode/formulacode-all/tree/897e48cab8a27d32ba20ddb970b1fc397d96ee95`, website repo `https://github.com/formula-code/formula-code.github.io/tree/a79001bfdf46cd6c6e11e742f8b98893294cbc7a`, eval repo `https://github.com/formula-code/fc-eval/tree/c08f665e7bf3b4de225b72dc02ce9b15b7aaba2b`. | Original scrape of 766 repositories, LLM environment synthesis, all statistical benchmark validations, and fresh agent runs. |
| Compute | Local CPU plus HF Job `https://huggingface.co/jobs/Srishti280992/6a6d1d066b79c09949c1dc23`; canary Job `https://huggingface.co/jobs/Srishti280992/6a6d1cf66b79c09949c1dc21`. | GPU/large-model inference for GPT-5, Claude 4.0 Sonnet, Gemini 2.5 Pro, and Qwen 3 Coder; likely many repo-specific benchmark runs. |
| Models linked | Public Hub model reference used in the paper family: `https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct`. Non-Hub APIs were not rerun. | Vendor API/model access plus the same Qwen model family, depending on the exact paper configuration. |
| Local outputs | `outputs/formulacode_audit/summary.json`, `summary.md`, `default_tasks_with_patch_stats.csv`, and raw first-party finding JSON copies. | Original raw benchmark measurements and per-task runtime distributions. |
| Estimated marginal cost | Under one CPU-basic HF Job plus local CPU time; effectively negligible for the audit. | Material API and compute cost for multi-agent benchmark execution. |
The reproduction poster was built with Posterly `https://github.com/gradio-app/posterly/tree/e503c399b5427ca6cb712ccb080a758e9c19cf23` and links each visual card to its logbook evidence page.
---
````html
````
### Fresh execution added by SabaPivot
I ran a separate CPU audit with seed `1082026`. It passed 4/4 checks. Independent arithmetic audit of the reported dataset funnel and benchmark summary; agent inference was not rerun. [Open the fresh audit](#/claim-99-fresh-independent-cpu-audit).
