# Evaluation Harness

This module instantiates a public task bundle into an isolated run directory,
runs one source-native Claude Code `QueryEngine` session, executes hidden judge
attempts, and writes compact JSONL trajectories for later skill extraction.

## Source-Native Architecture

The harness entrypoint calls `runSourceTaskLoop`, which creates exactly one `SourceClaudeSessionAgent` for the run. That session
constructs `QueryEngine` directly from the local Claude source and reuses it
across judge feedback rounds.

Per run:

1. `createTaskRun` copies only public task files into `runs/<run_id>/public`.
2. `resolveTaskRuntime` verifies the task Python before agent startup.
3. `SourceClaudeSessionAgent` builds a minimal tool pool from Claude source
   tools plus the in-process `finalize_submission` tool.
4. `createHarnessCanUseTool` enforces run-local read/write policy for tool use.
5. `DefaultJudgeRunner` copies private judge assets into `.judge_private` only
   when a judge attempt is triggered.

Batch mode is source-only: repeated `--task` values are dispatched as one worker
process per run. The removed legacy runtime is rejected explicitly.

## Loop Semantics

`max_rounds` means maximum judge attempts, not maximum agent tool calls. A judge
attempt is triggered only when the agent calls `finalize_submission` and output
validation passes against the public output contract. Validation failures are
returned as recoverable tool feedback and do not consume a judge round.

Natural-language messages such as "done" are ignored. If an agent turn ends
without a passing `finalize_submission`, the harness gives exactly one
same-round no-finalize recovery prompt. Recovery does not auto-validate,
auto-submit, or auto-judge; if it still ends without a passing
`finalize_submission`, the run fails or times out without a judge attempt.

Missing `workspace/plan.md` or `workspace/plans/round_NN.md` files are recorded
as warnings, not hard blockers, when outputs are otherwise valid.

## Task Visibility

Agent-visible paths are limited to the current run:

- `public/`: read-only public task bundle.
- `workspace/`: writable solver code, notes, and scratch files.
- `outputs/`: writable final submission files.
- `logs/agent/`: writable agent logs.

Private `evaluation/`, `std_code/`, reference output, and judge data paths are
not copied into the agent-visible workspace. They are available only to the
external judge under `runs/.judge_private/<run_id>/`.

The initial prompt now includes a compact public file manifest, output contract,
visible case summary, and the full public README. No `public/HARNESS_HINTS.md`
file is generated.

## Runtime Resolution

The harness resolves Python once before creating the Claude session. It reads
`public/envs/env_manifest.json`, chooses the platform-specific Python path, and
falls back to conventional runtime locations:

- Windows: `public/envs/runtime/.venv/Scripts/python.exe`
- POSIX/WSL: `public/envs/runtime/.venv/bin/python`
- POSIX/WSL alternate: `public/envs/runtime/.venv-posix/bin/python`

If no runtime exists, the run exits as `infra_error`; the agent is not asked to
repair virtual environments or install packages.

Virtual-environment directories are linked into the run instead of copied, so a
run does not spend minutes duplicating `site-packages`.

## Trajectories

`SourceTrajectoryWriter` writes two append-only files:

- `logs/trajectory.clean.jsonl`: compact training-friendly records.
- `logs/trajectory.raw.jsonl`: raw source events for debugging.

Static context is written once as `run_context`. Judge feedback is compacted
before being sent as a continuation prompt in the same session, while the raw
judge JSON stays in `trajectory.raw.jsonl` for debugging.

Clean trajectories also record validation pass/fail events, run warnings, and
no-finalize recovery markers. Raw trajectories preserve source events exactly;
clean trajectories filter standalone punctuation-only assistant text such as
`.` or `...`.

## CLI

```bash
bun src/harness/evaluation/cli.ts --task <task_id> --runs-dir output/runs --max-rounds 3 --timeout-seconds 1800 --temperature 1 --thinking disabled
```

`--system-prompt <path>` is a debug-only explicit override. Normal source-native
runs use the built-in lean system contract plus the generated initial prompt;
there is no default `config/AGENT.md`.

The default LLM options are `--thinking disabled --temperature 1`. With
thinking disabled, temperature is sent to the model. With `--thinking adaptive`,
temperature is intentionally omitted and logged as ignored in run metadata.

`--max-turns-per-round <n>` caps agentic turns within one judge round. It is
optional; full implementation regressions usually leave it unset and rely on
`--timeout-seconds` as the whole-loop hard timeout. Set a cap when diagnosing
no-finalize loops or running short smoke tests.