REPRODUCING.md

# Reproducing the agent-harness study

This repository separates inexpensive artifact verification from expensive
model-in-the-loop reruns. Start with the saved-evidence path; it validates the
paper without loading an LLM or embedding model.

## Requirements

- macOS on the recorded Apple Silicon host for an exact runtime replication,
  or Python 3.11+ on another platform for analysis-only reproduction.
- Git and Go 1.26.5 for reconstructing and validating benchmark tasks.
- LM Studio 0.4.16+2 and the exact model variants in `docs/LM_STUDIO.md` only
  when rerunning inference.
- No PyTorch workload is used. Any future Torch experiment must use MPS; CPU
  and CUDA runs are outside the repository's execution policy.

Create an analysis environment:

```bash
python3.11 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -e .
.venv/bin/python -m pip install -r requirements-analysis.txt
```

For exact dependency versions, install `requirements.lock` instead of the two
package-install commands above.

## Verify the archived evidence

The compact Zenodo archive contains all derived inputs required by these
commands:

```bash
PYTHONPATH=src .venv/bin/python -m agent_harness.cli validate
PYTHONPATH=src .venv/bin/python -m pytest -q
PYTHONPATH=src .venv/bin/python scripts/check_study2_paper_consistency.py
PYTHONPATH=src .venv/bin/python scripts/check_study5_paper_consistency.py
MPLBACKEND=Agg MPLCONFIGDIR=/tmp/agent-harness-mpl \
  PYTHONPATH=src .venv/bin/python scripts/analyze_study5.py
```

Earlier-study analysis commands and frozen revisions are listed in
`paper/reproducibility_manifest.md`. The fail-closed Study 5 analysis verifies
the exact E13--E16 cell grids, revisions, raw hashes, model-residency records,
required artifacts, and final outcome funnels before emitting results.

## Build the paper

```bash
mkdir -p tmp/pdfs
cd paper
latexmk -pdf -interaction=nonstopmode -halt-on-error \
  -outdir=../tmp/pdfs main.tex
cp ../tmp/pdfs/main.pdf \
  ../output/pdf/dissecting_repository_scale_code_agent_harnesses.pdf
```

Then verify the PDF against `output/pdf/SHA256SUMS` and inspect a PNG rendering
of every page. The publication package is rebuilt only after committing:

```bash
PYTHONPATH=src .venv/bin/python scripts/build_publication_package.py
```

## Rerun inference

Inference requires the ignored repository checkouts under `data/repos`, the
ignored indexes, and the exact local LM Studio models documented in
`docs/LM_STUDIO.md`. Run the matching preflight before every experiment family.
For Study 5:

```bash
PYTHONPATH=src .venv/bin/python scripts/preflight_study5.py
PYTHONPATH=src .venv/bin/python -m agent_harness.cli run-study5 --experiment E13
PYTHONPATH=src .venv/bin/python -m agent_harness.cli run-study5 --experiment E14
PYTHONPATH=src .venv/bin/python -m agent_harness.cli run-study5 --experiment E15
PYTHONPATH=src .venv/bin/python -m agent_harness.cli run-study5 --experiment E16
```

The runner uses `lms` for server lifecycle and official LM Studio APIs for
model inspection, load, and unload. Generative and embedding models are never
intentionally co-resident. Durable run identities make interruption recovery
idempotent.

## Artifact boundary

The compact archive excludes raw trajectories, repository checkouts, index
caches, and model weights. Raw trajectories should be uploaded as a separate
checksum-addressed Zenodo file or linked deposit. Some immutable provenance
records retain machine-local paths; these identify the original environment
and are not required paths for reproduction.