Hugging Face's logo

starpacker52
/
biodsbench-adapter

Model card Files
xet
 Community
biodsbench-adapter / src /harness /evaluation /README.md
starpacker52's picture
starpacker52
Add files using upload-large-folder tool
57c9ddf verified
preview code
|
Raw
History Blame Contribute Delete
4.88 kB

Evaluation Harness

This module instantiates a public task bundle into an isolated run directory, runs one source-native Claude Code QueryEngine session, executes hidden judge attempts, and writes compact JSONL trajectories for later skill extraction.

Source-Native Architecture

The harness entrypoint calls runSourceTaskLoop, which creates exactly one SourceClaudeSessionAgent for the run. That session constructs QueryEngine directly from the local Claude source and reuses it across judge feedback rounds.

Per run:

  1. createTaskRun copies only public task files into runs/<run_id>/public.
  2. resolveTaskRuntime verifies the task Python before agent startup.
  3. SourceClaudeSessionAgent builds a minimal tool pool from Claude source tools plus the in-process finalize_submission tool.
  4. createHarnessCanUseTool enforces run-local read/write policy for tool use.
  5. DefaultJudgeRunner copies private judge assets into .judge_private only when a judge attempt is triggered.

Batch mode is source-only: repeated --task values are dispatched as one worker process per run. The removed legacy runtime is rejected explicitly.

Loop Semantics

max_rounds means maximum judge attempts, not maximum agent tool calls. A judge attempt is triggered only when the agent calls finalize_submission and output validation passes against the public output contract. Validation failures are returned as recoverable tool feedback and do not consume a judge round.

Natural-language messages such as "done" are ignored. If an agent turn ends without a passing finalize_submission, the harness gives exactly one same-round no-finalize recovery prompt. Recovery does not auto-validate, auto-submit, or auto-judge; if it still ends without a passing finalize_submission, the run fails or times out without a judge attempt.

Missing workspace/plan.md or workspace/plans/round_NN.md files are recorded as warnings, not hard blockers, when outputs are otherwise valid.

Task Visibility

Agent-visible paths are limited to the current run:

  • public/: read-only public task bundle.
  • workspace/: writable solver code, notes, and scratch files.
  • outputs/: writable final submission files.
  • logs/agent/: writable agent logs.

Private evaluation/, std_code/, reference output, and judge data paths are not copied into the agent-visible workspace. They are available only to the external judge under runs/.judge_private/<run_id>/.

The initial prompt now includes a compact public file manifest, output contract, visible case summary, and the full public README. No public/HARNESS_HINTS.md file is generated.

Runtime Resolution

The harness resolves Python once before creating the Claude session. It reads public/envs/env_manifest.json, chooses the platform-specific Python path, and falls back to conventional runtime locations:

  • Windows: public/envs/runtime/.venv/Scripts/python.exe
  • POSIX/WSL: public/envs/runtime/.venv/bin/python
  • POSIX/WSL alternate: public/envs/runtime/.venv-posix/bin/python

If no runtime exists, the run exits as infra_error; the agent is not asked to repair virtual environments or install packages.

Virtual-environment directories are linked into the run instead of copied, so a run does not spend minutes duplicating site-packages.

Trajectories

SourceTrajectoryWriter writes two append-only files:

  • logs/trajectory.clean.jsonl: compact training-friendly records.
  • logs/trajectory.raw.jsonl: raw source events for debugging.

Static context is written once as run_context. Judge feedback is compacted before being sent as a continuation prompt in the same session, while the raw judge JSON stays in trajectory.raw.jsonl for debugging.

Clean trajectories also record validation pass/fail events, run warnings, and no-finalize recovery markers. Raw trajectories preserve source events exactly; clean trajectories filter standalone punctuation-only assistant text such as . or ....

CLI 

bun src/harness/evaluation/cli.ts --task <task_id> --runs-dir output/runs --max-rounds 3 --timeout-seconds 1800 --temperature 1 --thinking disabled

--system-prompt <path> is a debug-only explicit override. Normal source-native runs use the built-in lean system contract plus the generated initial prompt; there is no default config/AGENT.md.

The default LLM options are --thinking disabled --temperature 1. With thinking disabled, temperature is sent to the model. With --thinking adaptive, temperature is intentionally omitted and logged as ignored in run metadata.

--max-turns-per-round <n> caps agentic turns within one judge round. It is optional; full implementation regressions usually leave it unset and rely on --timeout-seconds as the whole-loop hard timeout. Set a cap when diagnosing no-finalize loops or running short smoke tests.