docs/STUDY2_IMPLEMENTATION.md

# Study 2 implementation contract

This document maps the preregistered E08 treatment names to executable behavior. It was frozen
before any Study 2 repair outcome was generated or inspected.

## Shared controls

Every main-matrix cell receives the same public task statement, 65,536-token LM Studio context,
production-file-only patch policy, frozen public-test allowlist, hidden-test patch, and
`resolved_at_1` evaluator. M002 and M003 run at temperature 0, top-p 1, seed 0, and at most 8,192
completion tokens per call. Interactive systems receive at most 12 model calls, 12 tool calls, two
public tests, and 1,800 seconds. The model never sees gold patches, hidden tests, gold symbols, or
gold files except that H018 deliberately receives gold file names as an upper-bound control.

Repository indexes are built at each task's base commit. Go uses Tree-sitter Go and `.go` source;
Python uses Tree-sitter Python and `.py` source. Dense chunks contain 120 lines, overlap by 20 lines,
and are capped at 16,000 characters. EMB002 supplies 1,024-dimensional normalized Qwen3 embedding
vectors. Its repository-neutral query instruction and all chunk/index parameters are hashed.

The embedding and agent models are never resident together. `lms server` controls the server;
official native REST endpoints load, inspect, and unload models. Each transition verifies one
exact model key and context. Agent discovery additionally verifies variant, format, quantization,
reasoning default, and inference-visible key.

## Component treatments

| ID | Search exposed to the LLM | Fusion/expansion | Purpose |
|---|---|---|---|
| H000 | literal, substring, or regular-expression scan | none | lexical floor |
| H003 | H000 plus dense retrieval | dense-first with exact backfill | embedding contribution |
| H007 | exact, BM25/fuzzy, Tree-sitter symbol, dense | reciprocal-rank fusion | full hybrid treatment |
| H011 | separate exact, lexical, syntax, dense, graph tools | one-hop symbol graph on fused seeds | specialized-interface/graph treatment |
| H018 | no search; gold production file names in task prompt | none | oracle-file upper bound |

Search returns at most five unique files with 24-line snippets. `read_file` returns at most 200
numbered lines. The agent edits only through a unified diff accepted by `git apply`. Accepted edits
are reconstructed relative to the frozen base before hidden evaluation.

## Controlled system baselines

`A001` is a local, controlled Agentless-style adaptation—not the authors' official Agentless
implementation. It performs exactly three independent model stages:

1. File localization from a deterministic repository path/symbol skeleton capped at 42,000 tokens.
2. Function/line localization from up to ten selected files, packed to 48,000 source tokens.
3. Unified-diff repair from the localized source and stage-2 output, followed by one frozen public
   validation command.

`A002` is a local, controlled SWE-agent-style adaptation—not the authors' official SWE-agent
implementation. It uses the same interactive loop and budgets as component agents but exposes an
agent-computer interface with `find_files`, literal/regex `search_text`, `read_file`, `apply_patch`,
`run_tests`, and `finish`. It does not use embeddings, Tree-sitter retrieval, or graph expansion.

These baselines test orchestration ideas while controlling the model, task, context, evaluator, and
local runtime. Results must be labeled “controlled adaptation”; they cannot be represented as
official Agentless or SWE-agent benchmark scores.

## Ordering, retries, and artifacts

The 60 tasks are balanced 20/20/20 across repositories. Model order alternates by task. Treatment
order rotates cyclically within each task. Raw cell identity hashes the experiment, task, treatment
configuration, model generation configuration, context, seed, repetition, repository SHA, and
research-code revision. Completed identities are immutable and resumed by reading their final
metrics. Incomplete infrastructure attempts are retained outside `raw/` before an identical retry.

The secondary reliability manifest freezes 24 non-oracle cells balanced across six treatments and
all six repository-by-model strata. They run under temperature 0.2 at seeds 0, 1, and 2 and use a
distinct configuration hash. These 72 cells are analyzed separately from the 840 deterministic
main cells.

Primary inspiration: [Agentless](https://arxiv.org/abs/2407.01489) and
[SWE-agent](https://arxiv.org/abs/2405.15793). Broader system context comes from
[OpenHands](https://arxiv.org/abs/2407.16741) and
[Multi-SWE-bench](https://arxiv.org/abs/2504.02605).