docs/STUDY2_PREREGISTRATION.md

# Study 2 Prospective Preregistration

**Frozen-design date:** 2026-07-18  
**Outcome-data status at freeze:** no Study 2 LLM repair outcome has been generated or inspected.  
**Purpose:** test whether repository-navigation harness effects replicate across repositories,
languages, and model families, and compare the component treatments with controlled adaptations
of established coding-agent orchestration systems.

## Research questions and claims

The confirmatory primary question is whether full hybrid retrieval (`H007`) improves end-to-end
repair success over exact/regular-expression search (`H000`) for the originally specified local
Qwen3.6-35B-A3B agent. The smallest effect of scientific interest is a 20 percentage-point paired
improvement. All other treatment and model-family conclusions are prespecified secondary claims.

The study will distinguish three claim types:

1. **Component causality:** comparisons among immutable harness configurations.
2. **System comparison:** controlled local reimplementations of published orchestration designs.
3. **Generalization:** variation across repositories, languages, and model families.

`A001` and `A002` must be described as controlled adaptations, not as results from the authors'
official implementations. The design is informed by Agentless, SWE-agent, OpenHands, and
Multi-SWE-bench, but does not claim implementation identity with those projects.

## Experimental units and benchmark construction

The frozen target sample is 60 real, test-backed software changes: 20 from each repository below.
All repositories are public GitLab checkouts, and every task is pinned to a full base and gold SHA.

| ID | Repository | Language | Head observed before sampling |
|---|---|---:|---|
| R001 | `gitlab-org/gitlab-runner` | Go | `c2831b75a3ff0782dca8f64498cbc6f71c76819e` |
| R002 | `gitlab-org/cli` | Go | `757294c01143360b466a70daf3fbef869fc3a41b` |
| R003 | `python-gitlab/python-gitlab` | Python | `f62dda7fa44e3bc46f03bd6402eba3f641f365eb` |

Ten previously frozen and validated R001 tasks may be reused. The remaining tasks are selected by
a deterministic, outcome-blind pipeline. Starting from the observed repository head and walking
first-parent history newest to oldest (not earlier than 2023-01-01), retain a change only if it:

- changes at least one production source file and at least one test file;
- changes no more than eight production files and eight test files;
- has no more than 400 changed production lines (`added + deleted`);
- is not dependency-only, formatting-only, generated-code-only, documentation-only, or a revert;
- yields disjoint production and hidden-test patches that both apply at the pinned base;
- has passing selected public tests at the base;
- has at least one held-out test that fails with only the test patch applied; and
- passes the same tests after applying the gold production patch.

Selection continues until 20 valid tasks exist for a repository. Rejections and their first failed
criterion are retained in a machine-readable audit. Task statements use the public issue/MR text
when available; otherwise they use the non-solution commit title/body. Statements exclude gold
paths, gold symbols, diff text, and test assertions. No LLM outcome may influence task selection.

The complete split, patches, validation logs, repository sizes, language counts, and task-difficulty
descriptors are frozen in Git before inference begins.

## Models and local runtime

Both agent models are served locally on port 1234 through LM Studio. `lms server` is the sole server
lifecycle controller. The native REST endpoints `/api/v1/models`, `/api/v1/models/load`, and
`/api/v1/models/unload` are the sole model residency controls. At most one model is resident at any
time.

| ID | Frozen model | LM Studio variant | Context | Reasoning |
|---|---|---|---:|---|
| M002 | Qwen3.6-35B-A3B | `qwen/qwen3.6-35b-a3b@4bit` (MLX 4-bit) | 65,536 | on |
| M003 | GPT-OSS-20B | `openai/gpt-oss-20b@mxfp4` (MLX MXFP4) | 65,536 | low |

Generation parameters are temperature 0, top-p 1, seed 0, and at most 8,192 completion tokens per
turn. The Qwen3 Embedding 0.6B Q8_0 model remains the fixed dense retriever. Embedding construction
and query inference occur under exclusive residency; the embedding model is unloaded before an
agent model is loaded. A wrong model key, variant, quantization, context, or reasoning mode is fatal.

## Treatments and execution budget

The component block contains five immutable harnesses:

| Treatment | Role |
|---|---|
| H000 | exact/substring/regex raw-file baseline |
| H003 | H000 plus dense code embeddings |
| H007 | exact, BM25/fuzzy, Tree-sitter, and dense retrieval with RRF |
| H011 | specialized retrieval tools plus one graph hop |
| H018 | oracle-file upper-bound control; file names only, never gold code |

The separate system block contains `A001`, a three-stage Agentless-style controlled adaptation,
and `A002`, a SWE-agent-style controlled interactive search/read/edit/test adaptation. Every
treatment uses the same frozen model build, task statement, context cap, patch restrictions, and
hidden-test evaluator. Interactive treatments receive at most 12 model turns, 12 tool calls, and
two public test runs. `A001` receives its native fixed three model stages and one validation run;
resource differences are reported rather than concealed.

The main matrix is `(5 harnesses + 2 systems) × 60 tasks × 2 models = 840` unique cells. Treatment
order is cyclically counterbalanced within repository/model blocks. Model order alternates by task.
An additional reliability audit repeats 24 deterministically selected non-oracle cells at seeds 1
and 2, balanced across the two models, three repositories, and six non-oracle treatments. This adds
48 cells, for 888 planned live cells total. Selection of repeat cells is frozen before primary
outcomes are read.

## Outcomes

The primary endpoint is `resolved_at_1`: the generated production patch applies, all held-out
fail-to-pass tests pass, and all pass-to-pass tests pass. No human patch repair is allowed.

Prespecified secondary outcomes are fail-to-pass success, pass-to-pass success, gold-file recall in
search/read/edit actions, exact modified-file match, patch applicability, empty-patch rate, model
turns, tool calls, test runs, latency, prompt/completion tokens, model-switch count/time, and
protocol-violation rate. Repository size in bytes, tracked source files, source lines, and estimated
tokens is reported to verify that full-repository prompting exceeds the 65,536-token treatment cap.

## Power and statistical analysis

The sole confirmatory primary contrast is H007 versus H000 on the 60 Qwen (`M002`) paired tasks.
It uses a two-sided exact McNemar test at alpha 0.05. Under preregistered discordance probabilities
`P(H007 only resolves)=0.25` and `P(H000 only resolves)=0.05`, the exact enumerated power is 0.797.
This calculation is reproduced by:

```bash
python3 scripts/power_study2.py --tasks 60 --new-only 0.25 --baseline-only 0.05 --alpha 0.05
```

The paired risk difference is reported with a task-cluster bootstrap 95% interval. The GPT-OSS
replication of H007 versus H000, the H003 contribution, H011 comparison, oracle gap, and each
controlled-system comparison are secondary. Exact paired p-values are Holm-adjusted within the
declared secondary family. Estimates and intervals are emphasized even when tests are nonsignificant.

A hierarchical logistic model includes fixed effects for treatment, model, repository/language,
and treatment-by-model interaction with a random task intercept. If that model is singular or fails
to converge, the preregistered fallback is a task-clustered GEE; if unavailable, a stratified
task-cluster bootstrap is reported without asymptotic model p-values. Reliability uses within-cell
agreement, variance of binary success, and intraclass agreement across the three seeds.

## Missingness, retries, exclusions, and stopping

Gold-validation failures are handled only before the task split is frozen. After freeze, a model
stop, malformed tool call, invalid patch, empty patch, timeout, or exhausted budget is an observed
failure, not missing data. An inference transport failure may be retried once only when no valid
assistant response was observed; the recovery is logged. Infrastructure failures that prevent any
cell observation are rerun under the identical immutable identity and reported. Cells are never
excluded based on success, patch quality, or treatment ranking.

Execution stops only after all 888 cells and declared analyses complete, or for a documented safety
condition such as insufficient disk, unavailable LM Studio runtime, irrecoverable repository test
environment, or unavailable MPS for any newly introduced Torch operation. This study does not use
Torch and does not invoke cloud infrastructure.

## Amendment policy and references

Any change after this document's first Git commit but before outcome inspection is recorded in
`docs/PROTOCOL_AMENDMENTS.md`, with its reason and whether it changes a claim. Any change after an
outcome is observed is labeled exploratory. The execution revision and dirty-tree check are written
to every run manifest.

Primary design references:

- Agentless: https://arxiv.org/abs/2407.01489
- SWE-agent: https://arxiv.org/abs/2405.15793
- OpenHands: https://arxiv.org/abs/2407.16741
- Multi-SWE-bench: https://arxiv.org/abs/2504.02605