Study 2 implementation contract
This document maps the preregistered E08 treatment names to executable behavior. It was frozen before any Study 2 repair outcome was generated or inspected.
Shared controls
Every main-matrix cell receives the same public task statement, 65,536-token LM Studio context,
production-file-only patch policy, frozen public-test allowlist, hidden-test patch, and
resolved_at_1 evaluator. M002 and M003 run at temperature 0, top-p 1, seed 0, and at most 8,192
completion tokens per call. Interactive systems receive at most 12 model calls, 12 tool calls, two
public tests, and 1,800 seconds. The model never sees gold patches, hidden tests, gold symbols, or
gold files except that H018 deliberately receives gold file names as an upper-bound control.
Repository indexes are built at each task's base commit. Go uses Tree-sitter Go and .go source;
Python uses Tree-sitter Python and .py source. Dense chunks contain 120 lines, overlap by 20 lines,
and are capped at 16,000 characters. EMB002 supplies 1,024-dimensional normalized Qwen3 embedding
vectors. Its repository-neutral query instruction and all chunk/index parameters are hashed.
The embedding and agent models are never resident together. lms server controls the server;
official native REST endpoints load, inspect, and unload models. Each transition verifies one
exact model key and context. Agent discovery additionally verifies variant, format, quantization,
reasoning default, and inference-visible key.
Component treatments
| ID | Search exposed to the LLM | Fusion/expansion | Purpose |
|---|---|---|---|
| H000 | literal, substring, or regular-expression scan | none | lexical floor |
| H003 | H000 plus dense retrieval | dense-first with exact backfill | embedding contribution |
| H007 | exact, BM25/fuzzy, Tree-sitter symbol, dense | reciprocal-rank fusion | full hybrid treatment |
| H011 | separate exact, lexical, syntax, dense, graph tools | one-hop symbol graph on fused seeds | specialized-interface/graph treatment |
| H018 | no search; gold production file names in task prompt | none | oracle-file upper bound |
Search returns at most five unique files with 24-line snippets. read_file returns at most 200
numbered lines. The agent edits only through a unified diff accepted by git apply. Accepted edits
are reconstructed relative to the frozen base before hidden evaluation.
Controlled system baselines
A001 is a local, controlled Agentless-style adaptation—not the authors' official Agentless
implementation. It performs exactly three independent model stages:
- File localization from a deterministic repository path/symbol skeleton capped at 42,000 tokens.
- Function/line localization from up to ten selected files, packed to 48,000 source tokens.
- Unified-diff repair from the localized source and stage-2 output, followed by one frozen public validation command.
A002 is a local, controlled SWE-agent-style adaptation—not the authors' official SWE-agent
implementation. It uses the same interactive loop and budgets as component agents but exposes an
agent-computer interface with find_files, literal/regex search_text, read_file, apply_patch,
run_tests, and finish. It does not use embeddings, Tree-sitter retrieval, or graph expansion.
These baselines test orchestration ideas while controlling the model, task, context, evaluator, and local runtime. Results must be labeled “controlled adaptation”; they cannot be represented as official Agentless or SWE-agent benchmark scores.
Ordering, retries, and artifacts
The 60 tasks are balanced 20/20/20 across repositories. Model order alternates by task. Treatment
order rotates cyclically within each task. Raw cell identity hashes the experiment, task, treatment
configuration, model generation configuration, context, seed, repetition, repository SHA, and
research-code revision. Completed identities are immutable and resumed by reading their final
metrics. Incomplete infrastructure attempts are retained outside raw/ before an identical retry.
The secondary reliability manifest freezes 24 non-oracle cells balanced across six treatments and all six repository-by-model strata. They run under temperature 0.2 at seeds 0, 1, and 2 and use a distinct configuration hash. These 72 cells are analyzed separately from the 840 deterministic main cells.
Primary inspiration: Agentless and SWE-agent. Broader system context comes from OpenHands and Multi-SWE-bench.