docs/STUDY4_PREREGISTRATION.md

# Study 4 prospective preregistration: protocol-normalized fresh-task retrieval

**Design date:** 2026-07-19  
**Outcome status at freeze:** no E10 model response or endpoint exists.  
**Sequential dependency:** E09's resolution-blind compatibility gate was frozen before E10 task
outcomes and selects P002 for M002, P003 for M003, and P003 for M004.

## Research question

E10 asks whether a full hybrid code-retrieval harness improves real repair performance over exact
raw search once each local LLM receives an edit action it can reliably express. The causal factor
is retrieval treatment:

- H000: exact path/substring/regular-expression retrieval over raw code;
- H007: exact plus BM25/fuzzy, Tree-sitter symbol, and dense code-embedding retrieval, fused by
  reciprocal rank fusion; and
- H018: gold production-file names only, an upper-bound localization control.

All arms share issue statement, frozen repository snapshot, model-specific E09-gated edit tool,
read/test/finish tools, test allowlist, hidden evaluator, decoding, 65,536-token model load, and
interaction budgets. H000 and H007 expose the same single `search_code` signature; only its fixed
retrieval implementation differs. H018 exposes no search and reveals only gold production-file
names, never gold code, symbols, locations, tests, or patch.

## Fresh task sample

The outcome-blind selector scanned reverse-chronological first-parent history since 2023 and
excluded every commit already in Studies 1--3. A candidate had to contain eligible production and
unit-test changes, pass its selected tests at the parent, have hidden tests that fail at the parent,
and pass after the gold production patch. Candidate decisions are retained under
`tasks/selection/study4/`.

The frozen split has 20 fresh tasks: three GitLab Runner (Go), seven GitLab CLI (Go), and ten
python-gitlab (Python). The first two histories yielded only three and seven valid fresh candidates
after exhausting the declared history window; no criterion was relaxed to force balance. Repository
and language effects are descriptive because repository and language remain partly confounded.

The main matrix is 20 tasks x 3 models x 3 retrieval treatments = **180 cells**. Every model-task
pair receives every retrieval treatment, and task-index rotations counterbalance treatment and
model order within the available unbalanced repository blocks.

## Endpoints and hypotheses

The sole confirmatory contrast is H007 versus H000 on `resolved_at_1`, pooled across the three
gate-eligible models. The estimand is the mean paired risk difference over the 60 model-task
strata, while uncertainty and randomization-style inference cluster the three model observations
within each of 20 tasks.

1. **H1 (confirmatory):** H007 increases `resolved_at_1` over H000.
2. **H2:** H007 increases search and read localization of at least one gold file before the first
   accepted edit.
3. **H3:** retrieval effects vary by model and repository.
4. **H4:** H018 provides an empirical localization upper bound but need not maximize resolution.

Secondary binary endpoints are accepted-edit cell, applicable final patch, any gold file in search
results, any gold file read before first edit, exact modified-file match, fail-to-pass success, and
resolution by model. Continuous process outcomes are calls, tests, tokens, latency, unique paths
searched/read, retrieval latency, embedding/model switches, and context compactions.

## Confirmatory inference and multiplicity

For H1, calculate each task's mean of the three paired H007-minus-H000 resolution differences.
Report the equally weighted mean across tasks, a 20,000-draw task-cluster bootstrap percentile 95%
interval (seed 20260720), and a two-sided exact task-cluster sign-flip test enumerating all
`2^20` assignments. This treats task, not model-task, as the independent sampling unit. Alpha is
0.05. The confirmatory result is reported regardless of direction or significance.

Prespecified secondary paired binary contrasts use exact McNemar tests within model where the 20
tasks are independent. All H007/H000 secondary p values, including three model-specific resolution
tests, form one Holm-adjusted family. Oracle contrasts, hierarchical models, and process metrics are
estimation-oriented and cannot replace H1.

A Bayesian binomial task-random-intercept model includes retrieval, model, repository,
retrieval-by-model, and task random intercept. Sparse or nonconvergent fits are reported as such.

## Mechanism and mediation

The registered mechanism chain is retrieval treatment -> gold file retrieved -> gold file read
before editing -> executor-accepted edit -> applicable final patch -> hidden-test resolution.
Stage-specific paired effects and transition rates will be reported. Because localization and edit
acceptance are post-treatment mediators, no untestable natural-direct/indirect causal effect will be
claimed. A nested logistic/g-computation mediation estimate may be shown only as a labeled
exploratory sensitivity analysis with its sequential-ignorability limitation explicit.

## Reliability, context scarcity, and robustness

After the 180 main cells are sealed, deterministic manifests select balanced non-oracle cells for
temperature-0.2 seeds 0/1/2 and a Qwen3.6 H000/H007 subset for 16,384 versus 65,536 loaded context.
These are prespecified sensitivity studies and are never pooled into H1. Feasibility is governed by
exclusive local model residency and memory safety; an ancillary block may be reduced only before
its first response, with the final manifest and reason retained. Previously frozen E04 robustness
evidence remains part of the integrated paper and is not relabeled as fresh E10 evidence.

## Missingness, execution, and stopping

All model/protocol failures are scored. A transport failure may be retried once only before a valid
assistant response; incomplete infrastructure attempts are archived outside scored cells. No cell
is excluded for poor outcome. Model and embedding models are loaded exclusively and sequentially
through the official LM Studio API; `lms` alone controls the server. The study uses no Torch and no
cloud resources. Main execution stops only after 180 finalized cells or a documented safety/runtime
blocker. Code, task split, gate, analysis, and conformance tests are committed before inference.