# Study 5 prospective preregistration: end-to-end harness behavior **Freeze rule:** this document, the E13--E15 experiment specifications, cell manifests, implementation tests, and design audit are committed before the first E13, E14, or E15 model response. Existing E01--E12 outcomes motivate the design but are not relabeled as prospective Study 5 evidence. ## Scientific objective Study 5 changes the paper's center of gravity from a single exact-versus-hybrid retrieval contrast to the behavior of repository-scale code-agent harnesses. It asks which declared retrieval, navigation, context, and action-interface choices change localization, accepted edits, applicable patches, hidden-test resolution, and systems cost, and whether those effects depend on the model. The study does not assume a universally best harness. Its primary output is a stage-aware effect atlas and a set of quality--cost Pareto candidates. ## Prospective operational correction Earlier live protocol runners always returned ranked snippets, permitted repeated search calls, and exposed one unified search signature in the protocol-normalized path. Those runs remain immutable and valid for contrasts that held these behaviors fixed. Study 5 prospectively operationalizes the previously catalogued dimensions: - `one_shot` permits one repository-search action; `iterative` permits repeated focused search actions within the common 12-tool budget; - `unified` exposes one fused search action; `specialized` exposes exact, lexical, syntax, dense, and graph search actions separately; and - search observations use ranked snippets, syntax skeletons, bounded whole files, or deterministic role summaries as declared by the harness file. This correction is frozen before new outcomes and is not applied retroactively. ## Experiments ### E13: retrieval-component factorial E13 crosses all eight combinations of advanced lexical retrieval (`L`), Tree-sitter syntax (`S`), and dense code embeddings (`D`) on the 60-task Study 2 panel and three local model families. Graph depth is zero, query policy is one-shot, the search interface is unified, packing uses ranked snippets, and the edit action is selected by the frozen E09 resolution-blind compatibility gate. The exact grid is 60 tasks x 8 harnesses x 3 models = **1,440 cells**. The primary E13 endpoint is resolved@1. The primary factorial estimands are the three marginal main effects averaged over the other sources. The prespecified secondary family contains the three pairwise interactions, the three-way interaction, accepted-edit and applicable-patch analogues, stage transitions, and model interactions. Resolution inference clusters on task; all cells for a task are paired. ### E14: retrieval by action interface E14 crosses exact retrieval (H000), the best prior static component combination (H006 syntax+dense), and full hybrid retrieval (H007) with raw unified diff (P001), exact replacement (P002), and whole-file replacement (P003), on the 20 Study 4 tasks and all three models. The exact grid is 20 x 3 x 3 x 3 = **540 cells**. The primary E14 estimand is the retrieval-by-action interaction on executor- accepted edits. Resolution and applicable-patch interactions are secondary. Model-by-action and model-by-retrieval interactions are reported because E09 established model-specific action compatibility. ### E15: navigation and packing blocks E15 evaluates H007--H015 on the 20 Study 4 tasks and three models using the E09 gate-selected action. The exact grid is 20 x 9 x 3 = **540 cells**. Prespecified paired blocks are: - graph: H007 versus H008 versus H009; - query policy: H008 versus H010; - search interface: H008 versus H011; - query by interface: H008, H010, H011, H012; and - packing: H008, H013, H014, H015. The primary endpoint for each block is resolved@1; accepted edit, applicable patch, gold retrieval/read before first accepted edit, token count, model time, retrieval/model-switch time, wall time, and protocol violations are secondary. Each block is corrected separately with Holm's method. ## Common protocol All cells use pinned GitLab commits, hidden fail-to-pass and pass-to-pass tests, temperature zero, seed zero, 65,536 loaded context, 12 tool calls, two public-test runs, the same system/task prompt apart from declared treatment descriptions, and exclusive LM Studio model residency. The Qwen3 Embedding 0.6B model is loaded only for index/query work and unloaded before a large LLM is loaded. Model loading, unloading, identity, quantization, context, prompts, responses, tools, edits, tests, tokens, timing, and memory telemetry are retained per immutable run identity. ## Multiplicity and uncertainty The task is the independent sampling unit. Binary paired contrasts use exact McNemar tests; factorial binary outcomes use a task-random-intercept logistic model with task-cluster bootstrap risk differences. Continuous paired outcomes use task-cluster bootstrap intervals and exact sign-flip tests where feasible. Twenty-thousand bootstrap draws use a frozen seed. Confirmatory families are Holm-adjusted. Seeds are never treated as independent tasks. Because prior resolution was sparse, a zero or near-zero difference is reported as no evidence of improvement, not equivalence, unless a separate equivalence margin and powered test are frozen before the relevant validation experiment. ## Sequential held-out validation After E13--E15 are complete, a deterministic selection script may choose at most six Pareto/archetype harnesses using a frozen rule: always retain H000; retain the highest-resolution treatment; retain the lowest-token treatment within five percentage points of that resolution; retain the highest accepted-edit treatment; retain the best prespecified robustness candidate; then fill remaining positions by harness ID from the nondominated quality--cost frontier. Ties break by applicable-patch rate, accepted-edit rate, lower mean tokens, lower mean wall time, then identifier. A separately mined outcome-blind task split and a new E16 manifest must be committed before any E16 response. E16 is confirmatory for rank stability, not for the screening p-values. ## Claim boundaries - E13--E15 reuse previously validated task panels but create new treatment cells; they estimate treatment behavior on those panels, not wholly new-task external validity. - E16, if enough additional valid tasks can be mined from pinned repository histories, provides the fresh-task check. - A harmful graph result applies to the tested expansion/reranking policy. - A model-specific interface effect is not an intrinsic model-quality ranking. - Pareto dominance is reported by outcome and cost; no post-hoc weighted omnibus score will be invented. ## E16 sequential freeze After E13--E15 completed, `scripts/freeze_study5_e16.py` applied the sequential rule above before any E16 model response. E13 supplies the primary screening panel for H000--H007 and E15 supplies it for H008--H015; H007 is not pooled a second time through its E15 anchor cells. E14 is excluded from archetype selection because its retrieval harnesses are deliberately repeated across three action interfaces and remain reserved for retrieval-by-action inference. The six frozen archetypes are H000 (mandatory exact baseline), H007 (highest resolution after the prespecified tie-break), H002 (lowest tokens within five percentage points of the best resolution), H014 (highest accepted-edit rate among remaining treatments), H008 (best of the prespecified H008/H010 robustness candidates), and H001 (lowest-ID remaining nondominated frontier member). The E16 grid is 17 fresh tasks x 6 harnesses x 3 models = **306 cells**, with the E09 model-specific action gate. The outcome-aware screening ledger hashes, role decisions, frontier, and selection hash are frozen in `configs/study5/E16_selection.json`; the E16-outcome-blind cells and manifest hash are frozen in `configs/study5/E16_cells.json`.