| # Metrics and artifacts |
|
|
| ## Outcome hierarchy |
|
|
| The primary end-to-end metric is `resolved_at_1`: one bounded agent run produces |
| a patch that applies, passes all fail-to-pass tests, and introduces no failure |
| in the pass-to-pass suite. The primary retrieval metric is file recall@10; |
| function recall@10, MRR, and NDCG@10 are secondary retrieval outcomes. |
|
|
| Metric groups are registered in `src/agent_harness/metrics.py`: |
|
|
| | Group | Examples | |
| |---|---| |
| | Correctness | resolution, fail-to-pass, pass-to-pass, regressions | |
| | Localization | recall@k, MRR, NDCG, gold evidence within budget | |
| | Navigation | time/tokens/tool calls to first gold, repeated reads, churn | |
| | Editing | patch application, syntax, lint/build, patch size | |
| | Efficiency | input/output/tool tokens, calls, tests, wall time, throughput | |
| | Index | build/update time, query p50/p95, RAM, disk | |
| | Reliability | timeout, tool/parser/edit failures, failure stage | |
|
|
| All rates require explicit numerators, denominators, and exclusion counts in a |
| paper table. Token usage must distinguish model input, model output, cached |
| input when reported by the server, and tool-result text. A missing usage field |
| is missing data, not zero. |
|
|
| ## Run identity and lineage |
|
|
| Every run has a deterministic ID derived from: |
|
|
| - experiment, task, and harness IDs; |
| - complete harness and model-config hashes; |
| - resolved LM Studio model key; |
| - context budget, seed, and repetition; |
| - repository commit and research-code revision. |
|
|
| Artifacts live under: |
|
|
| ```text |
| results/raw/<experiment>/<harness>/<task>/<run_id>/ |
| run_manifest.json |
| trajectory.jsonl |
| patch.diff |
| test_results.json |
| final_metrics.json |
| ``` |
|
|
| The raw run directory is created exclusively and must never be overwritten. |
| `trajectory.jsonl` is append-only and contains monotonically sequenced events. |
| The manifest stores the resolved treatment and model metadata so a result never |
| depends only on a filename. |
|
|
| ## Required trajectory events |
|
|
| - `run_started`: limits, environment, task metadata, execution order |
| - `model_call`: request hash, sampling, token usage, latency, stop reason |
| - `tool_call`: tool name, sanitized arguments, result hash, size, latency |
| - `retrieval_candidate`: source, query, file/symbol span, score, rank, fusion |
| - `file_read`: path, span, content hash, token count, relevance label after run |
| - `edit`: patch hash, target paths, application result |
| - `test_run`: command identity, exit status, duration, structured test counts |
| - `resource_sample`: process memory and relevant local-system measurements |
| - `run_finished`: outcome, failure stage, totals, artifact hashes |
|
|
| Secrets, environment contents, and unrelated user files must never enter |
| telemetry. Large source snippets are identified by repository SHA and content |
| hash; record only the content needed to audit the model-visible context. |
|
|
| ## Derived datasets and tables |
|
|
| Raw JSONL is the source of truth. Deterministic analysis code should produce a |
| versioned tabular dataset with one row per run and separate candidate/query |
| tables. Planned paper tables are: |
|
|
| 1. task and repository characteristics; |
| 2. retrieval factorial effects and interactions; |
| 3. query/interface and packing ablations; |
| 4. end-to-end correctness and efficiency; |
| 5. robustness by scenario and difficulty stratum; |
| 6. dense-index systems results; and |
| 7. failure taxonomy and exclusions. |
|
|
| Every derived artifact should record the input run IDs, analysis-code revision, |
| schema version, and creation timestamp. |
|
|
|
|
