docs/STUDY5_PREREGISTRATION.md

# Study 5 prospective preregistration: end-to-end harness behavior

**Freeze rule:** this document, the E13--E15 experiment specifications, cell
manifests, implementation tests, and design audit are committed before the first
E13, E14, or E15 model response. Existing E01--E12 outcomes motivate the design
but are not relabeled as prospective Study 5 evidence.

## Scientific objective

Study 5 changes the paper's center of gravity from a single exact-versus-hybrid
retrieval contrast to the behavior of repository-scale code-agent harnesses. It
asks which declared retrieval, navigation, context, and action-interface choices
change localization, accepted edits, applicable patches, hidden-test resolution,
and systems cost, and whether those effects depend on the model.

The study does not assume a universally best harness. Its primary output is a
stage-aware effect atlas and a set of quality--cost Pareto candidates.

## Prospective operational correction

Earlier live protocol runners always returned ranked snippets, permitted repeated
search calls, and exposed one unified search signature in the protocol-normalized
path. Those runs remain immutable and valid for contrasts that held these behaviors
fixed. Study 5 prospectively operationalizes the previously catalogued dimensions:

- `one_shot` permits one repository-search action; `iterative` permits repeated
  focused search actions within the common 12-tool budget;
- `unified` exposes one fused search action; `specialized` exposes exact, lexical,
  syntax, dense, and graph search actions separately; and
- search observations use ranked snippets, syntax skeletons, bounded whole files,
  or deterministic role summaries as declared by the harness file.

This correction is frozen before new outcomes and is not applied retroactively.

## Experiments

### E13: retrieval-component factorial

E13 crosses all eight combinations of advanced lexical retrieval (`L`),
Tree-sitter syntax (`S`), and dense code embeddings (`D`) on the 60-task Study 2
panel and three local model families. Graph depth is zero, query policy is
one-shot, the search interface is unified, packing uses ranked snippets, and the
edit action is selected by the frozen E09 resolution-blind compatibility gate.
The exact grid is 60 tasks x 8 harnesses x 3 models = **1,440 cells**.

The primary E13 endpoint is resolved@1. The primary factorial estimands are the
three marginal main effects averaged over the other sources. The prespecified
secondary family contains the three pairwise interactions, the three-way
interaction, accepted-edit and applicable-patch analogues, stage transitions,
and model interactions. Resolution inference clusters on task; all cells for a
task are paired.

### E14: retrieval by action interface

E14 crosses exact retrieval (H000), the best prior static component combination
(H006 syntax+dense), and full hybrid retrieval (H007) with raw unified diff
(P001), exact replacement (P002), and whole-file replacement (P003), on the 20
Study 4 tasks and all three models. The exact grid is
20 x 3 x 3 x 3 = **540 cells**.

The primary E14 estimand is the retrieval-by-action interaction on executor-
accepted edits. Resolution and applicable-patch interactions are secondary.
Model-by-action and model-by-retrieval interactions are reported because E09
established model-specific action compatibility.

### E15: navigation and packing blocks

E15 evaluates H007--H015 on the 20 Study 4 tasks and three models using the E09
gate-selected action. The exact grid is 20 x 9 x 3 = **540 cells**. Prespecified
paired blocks are:

- graph: H007 versus H008 versus H009;
- query policy: H008 versus H010;
- search interface: H008 versus H011;
- query by interface: H008, H010, H011, H012; and
- packing: H008, H013, H014, H015.

The primary endpoint for each block is resolved@1; accepted edit, applicable
patch, gold retrieval/read before first accepted edit, token count, model time,
retrieval/model-switch time, wall time, and protocol violations are secondary.
Each block is corrected separately with Holm's method.

## Common protocol

All cells use pinned GitLab commits, hidden fail-to-pass and pass-to-pass tests,
temperature zero, seed zero, 65,536 loaded context, 12 tool calls, two public-test
runs, the same system/task prompt apart from declared treatment descriptions,
and exclusive LM Studio model residency. The Qwen3 Embedding 0.6B model is loaded
only for index/query work and unloaded before a large LLM is loaded. Model loading,
unloading, identity, quantization, context, prompts, responses, tools, edits, tests,
tokens, timing, and memory telemetry are retained per immutable run identity.

## Multiplicity and uncertainty

The task is the independent sampling unit. Binary paired contrasts use exact
McNemar tests; factorial binary outcomes use a task-random-intercept logistic
model with task-cluster bootstrap risk differences. Continuous paired outcomes
use task-cluster bootstrap intervals and exact sign-flip tests where feasible.
Twenty-thousand bootstrap draws use a frozen seed. Confirmatory families are
Holm-adjusted. Seeds are never treated as independent tasks.

Because prior resolution was sparse, a zero or near-zero difference is reported
as no evidence of improvement, not equivalence, unless a separate equivalence
margin and powered test are frozen before the relevant validation experiment.

## Sequential held-out validation

After E13--E15 are complete, a deterministic selection script may choose at most
six Pareto/archetype harnesses using a frozen rule: always retain H000; retain the
highest-resolution treatment; retain the lowest-token treatment within five
percentage points of that resolution; retain the highest accepted-edit treatment;
retain the best prespecified robustness candidate; then fill remaining positions
by harness ID from the nondominated quality--cost frontier. Ties break by
applicable-patch rate, accepted-edit rate, lower mean tokens, lower mean wall time,
then identifier. A separately mined outcome-blind task split and a new E16 manifest
must be committed before any E16 response. E16 is confirmatory for rank stability,
not for the screening p-values.

## Claim boundaries

- E13--E15 reuse previously validated task panels but create new treatment cells;
  they estimate treatment behavior on those panels, not wholly new-task external
  validity.
- E16, if enough additional valid tasks can be mined from pinned repository
  histories, provides the fresh-task check.
- A harmful graph result applies to the tested expansion/reranking policy.
- A model-specific interface effect is not an intrinsic model-quality ranking.
- Pareto dominance is reported by outcome and cost; no post-hoc weighted omnibus
  score will be invented.

## E16 sequential freeze

After E13--E15 completed, `scripts/freeze_study5_e16.py` applied the sequential
rule above before any E16 model response. E13 supplies the primary screening
panel for H000--H007 and E15 supplies it for H008--H015; H007 is not pooled a
second time through its E15 anchor cells. E14 is excluded from archetype
selection because its retrieval harnesses are deliberately repeated across
three action interfaces and remain reserved for retrieval-by-action inference.

The six frozen archetypes are H000 (mandatory exact baseline), H007 (highest
resolution after the prespecified tie-break), H002 (lowest tokens within five
percentage points of the best resolution), H014 (highest accepted-edit rate
among remaining treatments), H008 (best of the prespecified H008/H010
robustness candidates), and H001 (lowest-ID remaining nondominated frontier
member). The E16 grid is 17 fresh tasks x 6 harnesses x 3 models = **306
cells**, with the E09 model-specific action gate. The outcome-aware screening
ledger hashes, role decisions, frontier, and selection hash are frozen in
`configs/study5/E16_selection.json`; the E16-outcome-blind cells and manifest
hash are frozen in `configs/study5/E16_cells.json`.