Hugging Face's logo

cuber12
/
agent-harness

code-agents
software-engineering
llm-evaluation
reproducibility
Model card Files
xet
 Community
agent-harness / REPRODUCING.md
cuber12's picture
cuber12
Publish agent harness research code and paper artifacts
d61821a verified
preview code
|
Raw
History Blame Contribute Delete
3.59 kB

Reproducing the agent-harness study

This repository separates inexpensive artifact verification from expensive model-in-the-loop reruns. Start with the saved-evidence path; it validates the paper without loading an LLM or embedding model.

Requirements

  • macOS on the recorded Apple Silicon host for an exact runtime replication, or Python 3.11+ on another platform for analysis-only reproduction.
  • Git and Go 1.26.5 for reconstructing and validating benchmark tasks.
  • LM Studio 0.4.16+2 and the exact model variants in docs/LM_STUDIO.md only when rerunning inference.
  • No PyTorch workload is used. Any future Torch experiment must use MPS; CPU and CUDA runs are outside the repository's execution policy.

Create an analysis environment:

python3.11 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -e .
.venv/bin/python -m pip install -r requirements-analysis.txt

For exact dependency versions, install requirements.lock instead of the two package-install commands above.

Verify the archived evidence

The compact Zenodo archive contains all derived inputs required by these commands:

PYTHONPATH=src .venv/bin/python -m agent_harness.cli validate
PYTHONPATH=src .venv/bin/python -m pytest -q
PYTHONPATH=src .venv/bin/python scripts/check_study2_paper_consistency.py
PYTHONPATH=src .venv/bin/python scripts/check_study5_paper_consistency.py
MPLBACKEND=Agg MPLCONFIGDIR=/tmp/agent-harness-mpl \
  PYTHONPATH=src .venv/bin/python scripts/analyze_study5.py

Earlier-study analysis commands and frozen revisions are listed in paper/reproducibility_manifest.md. The fail-closed Study 5 analysis verifies the exact E13--E16 cell grids, revisions, raw hashes, model-residency records, required artifacts, and final outcome funnels before emitting results.

Build the paper 

mkdir -p tmp/pdfs
cd paper
latexmk -pdf -interaction=nonstopmode -halt-on-error \
  -outdir=../tmp/pdfs main.tex
cp ../tmp/pdfs/main.pdf \
  ../output/pdf/dissecting_repository_scale_code_agent_harnesses.pdf

Then verify the PDF against output/pdf/SHA256SUMS and inspect a PNG rendering of every page. The publication package is rebuilt only after committing:

PYTHONPATH=src .venv/bin/python scripts/build_publication_package.py

Rerun inference

Inference requires the ignored repository checkouts under data/repos, the ignored indexes, and the exact local LM Studio models documented in docs/LM_STUDIO.md. Run the matching preflight before every experiment family. For Study 5:

PYTHONPATH=src .venv/bin/python scripts/preflight_study5.py
PYTHONPATH=src .venv/bin/python -m agent_harness.cli run-study5 --experiment E13
PYTHONPATH=src .venv/bin/python -m agent_harness.cli run-study5 --experiment E14
PYTHONPATH=src .venv/bin/python -m agent_harness.cli run-study5 --experiment E15
PYTHONPATH=src .venv/bin/python -m agent_harness.cli run-study5 --experiment E16

The runner uses lms for server lifecycle and official LM Studio APIs for model inspection, load, and unload. Generative and embedding models are never intentionally co-resident. Durable run identities make interruption recovery idempotent.

Artifact boundary

The compact archive excludes raw trajectories, repository checkouts, index caches, and model weights. Raw trajectories should be uploaded as a separate checksum-addressed Zenodo file or linked deposit. Some immutable provenance records retain machine-local paths; these identify the original environment and are not required paths for reproduction.