| # Implementation status |
|
|
| ## Completed |
|
|
| - immutable Python dataclasses, TOML validation, and one separate configuration |
| file for each H000-H020 harness; |
| - strict official-API discovery, loading, unloading, and identity checks for |
| Qwen3.6-35B-A3B, GPT-OSS-20B, Qwen3-Coder-30B, and Qwen3 Embedding 0.6B; |
| - phase-exclusive LM Studio model residency controlled through `lms server` |
| plus native REST model lifecycle endpoints; |
| - frozen Git snapshots, exact/regex search, BM25/fuzzy retrieval, Tree-sitter |
| symbols, dense retrieval, rank fusion, graph expansion, context packers, and |
| unified/specialized interfaces; |
| - worktree-isolated patch application, hidden fail-to-pass/pass-to-pass test |
| execution, terminal failure staging, and append-only trajectories; |
| - FAISS FlatIP, FAISS HNSW, and sqlite-vec backend evaluation; |
| - 965 Study 1 cells, including 70 live Qwen tool-loop cells; |
| - a prospectively frozen Study 2 with 60 real-change tasks across GitLab Runner, |
| GitLab CLI, and python-gitlab; seven treatments; two local models; 840 main |
| cells; and 72 repeated-seed cells; |
| - artifact identity audits, paired exact tests, 20,000-sample task-cluster |
| bootstrap intervals, Holm correction, and Bayesian binomial task-random- |
| intercept analysis; |
| - a prospectively frozen Study 3 crossing 60 tasks, three local models, and |
| three edit interfaces in 540 real LLM cells, with an outcome-blind |
| compatibility gate for the fresh retrieval replication; |
| - a prospectively frozen Study 4 with 20 newly mined executable tasks, three |
| gated models, exact/hybrid/oracle retrieval, 180 main cells, 18 stochastic |
| reliability cells, and 12 context-scarcity cells; |
| - a prospectively frozen Study 5 with 1,440 retrieval-component factorial |
| cells, 540 retrieval-by-action cells, 540 navigation/packing cells, and a |
| separately frozen 306-cell held-out validation on 17 fresh tasks; |
| - a LaTeX manuscript, bibliography, generated evidence, reproducibility |
| manifests, and rendered-PDF visual QA; |
| - public-release metadata, dual licensing, third-party notices, a standalone |
| reproduction guide, and a fail-closed repository/Zenodo audit. |
|
|
| ## Evidence inventory |
|
|
| | Study | Experiment | Cells | Terminal scope | |
| |---|---|---:|---| |
| | 1 | E01 | 150 | Static retrieval and graph depth | |
| | 1 | E02 | 105 | Model-selected localization, interface, query, packing | |
| | 1 | E03 | 100 | Strict single-generation repair | |
| | 1 | E04 | 135 | Stale-index and nested-distractor robustness | |
| | 1 | E05 | 405 | Dense vector backends and timing repetitions | |
| | 1 | E07 | 70 | Live Qwen tool loop with hidden-test evaluation | |
| | 2 | E08 main | 840 | 60 tasks x 7 treatments x 2 models | |
| | 2 | E08 reliability | 72 | 24 groups x 3 stochastic seeds | |
| | 3 | E09 | 540 | Model-by-edit-interface compatibility | |
| | 4 | E10 | 180 | Fresh protocol-normalized retrieval replication | |
| | 4 | E11 | 18 | Six balanced groups x three stochastic seeds | |
| | 4 | E12 | 12 | Qwen exact/hybrid x 16K/65K context | |
| | 5 | E13 | 1,440 | Lexical/syntax/dense factorial x three models | |
| | 5 | E14 | 540 | Retrieval-by-edit-interface factorial | |
| | 5 | E15 | 540 | Graph/query/interface/packing paired blocks | |
| | 5 | E16 | 306 | Six archetypes x 17 fresh tasks x three models | |
| | **Total** | | **5,453** | Audited unique scored run IDs | |
|
|
| Development-only artifacts remain excluded: 15 E00 pilot cells, three |
| superseded E02 cells, and one E06 smoke cell. Frozen analysis is in |
| `results/derived/confirmatory_analysis_174ce71bcbce`, `results/derived/e07`, and |
| `results/derived/study2`, `results/derived/study3`, `results/derived/study4`, |
| `results/derived/study4_ancillary`, and `results/derived/study5`. The stable paper is |
| `output/pdf/dissecting_repository_scale_code_agent_harnesses.pdf`. |
|
|
| ## Scientific outcome |
|
|
| Study 2 did not establish a winning retrieval harness. The registered H007 vs |
| H000/Qwen contrast was 2/60 vs 2/60 (RD 0.000, CI [-0.050, 0.050], p=1), and |
| all corrected secondary tests were null. Study 3 then causally established |
| model-specific action-protocol effects. Its registered GPT-OSS contrast was |
| 27/60 accepted edit cells under exact replacement versus 0/60 under strict |
| unified diff (paired RD +0.450, 95% CI [0.333, 0.583], exact p=1.49e-8). |
| The outcome-blind gate selected P002 for M002 and P003 for M003/M004. This |
| separates executor compatibility from retrieval quality in the next fresh-task |
| replication. |
|
|
| Study 4 then tested retrieval on 20 fresh tasks under the gate. H007 resolved |
| 1/60 model-task pairs versus 0/60 for H000 (RD +0.0167, task-cluster 95% CI |
| [0, 0.05], exact sign-flip p=1). H007 improved no registered localization |
| stage and was slower. H018 resolved 4/60, showing that file knowledge helps |
| occasionally but does not remove the semantic repair bottleneck. E11 found |
| unanimous failure but zero unanimous trajectories across six groups; E12 found |
| no resolution difference between 16K and 65K context. |
|
|
| Study 5 then tested the harness design space directly. E13 found no corrected |
| lexical, syntax, or dense component main effect. E14 found no corrected |
| retrieval-by-action interaction. E15 showed that iteration and whole-file |
| packing increase systems cost without a demonstrated applicable-patch gain. |
| E16 quality ranks transferred weakly to 17 fresh tasks, while token-cost rank |
| transferred strongly; only one E16 cell resolved, so no universal winner is |
| claimed. |
|
|
| ## Deliberately outside this paper |
|
|
| - official, unmodified Agentless/SWE-agent/OpenHands implementations; |
| - larger repository and programming-language samples that independently |
| identify language effects; |
| - protocol-adaptive patch normalization as a confirmatory treatment; |
| - additional model sizes, quantizations, and context budgets; |
| - hidden-test-feedback retries or production call/diff repair. |
|
|
