docs/PAPER_PLAN.md

# Paper plan

Status: completed through E07. This file preserves the pre-execution plan; the
realized design, deviations, evidence inventory, and release gates are recorded
in `paper/main.tex` and `paper/reproducibility_manifest.md`.

## Working title

**Beyond Grep: A Controlled Study of Repository-Navigation Harnesses for Local
LLM Coding Agents**

## Candidate contribution

The paper should make a narrower, defensible claim than “better harnesses make
agents better.” The intended contribution is a controlled empirical
decomposition of navigation architecture—retrieval source, structural
expansion, interaction policy, tool interface, and context packing—under a
fixed local coding model on repositories larger than the model's usable
context.

Potential publishable outputs are:

1. an open, immutable harness-treatment catalog with focused factorial blocks;
2. paired evidence connecting localization quality to end-to-end repair;
3. accuracy/latency/token trade-off curves rather than success alone;
4. robustness results for stale indexes and plausible lexical distractors; and
5. a reproducible local inference and telemetry protocol.

Novelty depends on the completed literature review and should not be asserted
until adjacent agent, code-retrieval, repository-level repair, and tool-use
benchmarks have been systematically compared.

## Prespecified hypotheses

- H1: lexical, syntax, and dense retrieval each improve file recall over exact
  search, but their effects are not purely additive.
- H2: one-hop graph expansion improves multi-file task localization; two hops
  increase noise and cost more often than they improve resolution.
- H3: iterative querying improves hard-task localization but consumes more
  tokens and wall time.
- H4: structural skeleton packing outperforms whole-file packing at small fixed
  context budgets.
- H5: improved file/function recall mediates part, but not all, of the effect on
  repair success.
- H6: adaptive routing can approach full-stack accuracy with lower retrieval
  and context cost.

These hypotheses must be revised or preregistered before looking at the held-out
confirmatory results.

## Manuscript structure

1. **Abstract:** problem, controlled design, task/model scope, main effect sizes.
2. **Introduction:** why context overflow turns navigation into a systems and
   reasoning bottleneck; research questions and contributions.
3. **Related work:** code search and embeddings, repository-level program
   repair, LLM tool use, agent benchmarks, context selection, code graphs.
4. **Method:** treatment dimensions, fixed model, task construction, execution
   controls, metrics, and statistical analysis.
5. **Experiments:** repository/task statistics, E01-E05, compute and local
   serving details.
6. **Results:** prespecified contrasts, confidence intervals, cost-quality
   frontiers, subgroup and robustness analyses.
7. **Discussion:** mechanisms, practical recommendations, failure cases, and
   where complex harnesses do not pay for themselves.
8. **Threats and ethics:** contamination, generalization, annotation quality,
   local hardware/quantization effects, licensing, and energy/compute reporting.
9. **Reproducibility statement:** configs, commits, prompts, manifests, raw event
   schemas, analysis code, exclusions, and environment lockfiles.

## Milestones and release gates

1. Complete executable baseline H000 and validate task isolation.
2. Select repositories and construct a clean development task set.
3. Implement and unit-test retrieval sources, fusion, packing, and telemetry.
4. Freeze embedding query/chunk policies, index profiles, model variant, prompts, and budgets.
5. Run a small pilot; perform failure analysis and power simulation.
6. Freeze protocol, task split, configs, analysis, and exclusion rules.
7. Run confirmatory cells without treatment-specific intervention.
8. Reproduce tables from raw artifacts, then write and internally audit the
   paper.