Study 4 prospective preregistration: protocol-normalized fresh-task retrieval

Design date: 2026-07-19
Outcome status at freeze: no E10 model response or endpoint exists.
Sequential dependency: E09's resolution-blind compatibility gate was frozen before E10 task outcomes and selects P002 for M002, P003 for M003, and P003 for M004.

Research question

E10 asks whether a full hybrid code-retrieval harness improves real repair performance over exact raw search once each local LLM receives an edit action it can reliably express. The causal factor is retrieval treatment:

H000: exact path/substring/regular-expression retrieval over raw code;
H007: exact plus BM25/fuzzy, Tree-sitter symbol, and dense code-embedding retrieval, fused by reciprocal rank fusion; and
H018: gold production-file names only, an upper-bound localization control.

All arms share issue statement, frozen repository snapshot, model-specific E09-gated edit tool, read/test/finish tools, test allowlist, hidden evaluator, decoding, 65,536-token model load, and interaction budgets. H000 and H007 expose the same single search_code signature; only its fixed retrieval implementation differs. H018 exposes no search and reveals only gold production-file names, never gold code, symbols, locations, tests, or patch.

Fresh task sample

The outcome-blind selector scanned reverse-chronological first-parent history since 2023 and excluded every commit already in Studies 1--3. A candidate had to contain eligible production and unit-test changes, pass its selected tests at the parent, have hidden tests that fail at the parent, and pass after the gold production patch. Candidate decisions are retained under tasks/selection/study4/.

The frozen split has 20 fresh tasks: three GitLab Runner (Go), seven GitLab CLI (Go), and ten python-gitlab (Python). The first two histories yielded only three and seven valid fresh candidates after exhausting the declared history window; no criterion was relaxed to force balance. Repository and language effects are descriptive because repository and language remain partly confounded.

The main matrix is 20 tasks x 3 models x 3 retrieval treatments = 180 cells. Every model-task pair receives every retrieval treatment, and task-index rotations counterbalance treatment and model order within the available unbalanced repository blocks.

Endpoints and hypotheses

The sole confirmatory contrast is H007 versus H000 on resolved_at_1, pooled across the three gate-eligible models. The estimand is the mean paired risk difference over the 60 model-task strata, while uncertainty and randomization-style inference cluster the three model observations within each of 20 tasks.

H1 (confirmatory): H007 increases resolved_at_1 over H000.
H2: H007 increases search and read localization of at least one gold file before the first accepted edit.
H3: retrieval effects vary by model and repository.
H4: H018 provides an empirical localization upper bound but need not maximize resolution.

Secondary binary endpoints are accepted-edit cell, applicable final patch, any gold file in search results, any gold file read before first edit, exact modified-file match, fail-to-pass success, and resolution by model. Continuous process outcomes are calls, tests, tokens, latency, unique paths searched/read, retrieval latency, embedding/model switches, and context compactions.

Confirmatory inference and multiplicity

For H1, calculate each task's mean of the three paired H007-minus-H000 resolution differences. Report the equally weighted mean across tasks, a 20,000-draw task-cluster bootstrap percentile 95% interval (seed 20260720), and a two-sided exact task-cluster sign-flip test enumerating all 2^20 assignments. This treats task, not model-task, as the independent sampling unit. Alpha is 0.05. The confirmatory result is reported regardless of direction or significance.

Prespecified secondary paired binary contrasts use exact McNemar tests within model where the 20 tasks are independent. All H007/H000 secondary p values, including three model-specific resolution tests, form one Holm-adjusted family. Oracle contrasts, hierarchical models, and process metrics are estimation-oriented and cannot replace H1.

A Bayesian binomial task-random-intercept model includes retrieval, model, repository, retrieval-by-model, and task random intercept. Sparse or nonconvergent fits are reported as such.

Mechanism and mediation

The registered mechanism chain is retrieval treatment -> gold file retrieved -> gold file read before editing -> executor-accepted edit -> applicable final patch -> hidden-test resolution. Stage-specific paired effects and transition rates will be reported. Because localization and edit acceptance are post-treatment mediators, no untestable natural-direct/indirect causal effect will be claimed. A nested logistic/g-computation mediation estimate may be shown only as a labeled exploratory sensitivity analysis with its sequential-ignorability limitation explicit.

Reliability, context scarcity, and robustness

After the 180 main cells are sealed, deterministic manifests select balanced non-oracle cells for temperature-0.2 seeds 0/1/2 and a Qwen3.6 H000/H007 subset for 16,384 versus 65,536 loaded context. These are prespecified sensitivity studies and are never pooled into H1. Feasibility is governed by exclusive local model residency and memory safety; an ancillary block may be reduced only before its first response, with the final manifest and reason retained. Previously frozen E04 robustness evidence remains part of the integrated paper and is not relabeled as fresh E10 evidence.

Missingness, execution, and stopping

All model/protocol failures are scored. A transport failure may be retried once only before a valid assistant response; incomplete infrastructure attempts are archived outside scored cells. No cell is excluded for poor outcome. Model and embedding models are loaded exclusively and sequentially through the official LM Studio API; lms alone controls the server. The study uses no Torch and no cloud resources. Main execution stops only after 180 finalized cells or a documented safety/runtime blocker. Code, task split, gate, analysis, and conformance tests are committed before inference.