agent-harness / docs /STUDY3_IMPLEMENTATION.md
cuber12's picture
Publish agent harness research code and paper artifacts
d61821a verified
|
Raw
History Blame Contribute Delete
1.2 kB

Study 3 implementation contract

E09 implements the prospective protocol in STUDY3_PREREGISTRATION.md. Its 540 cells reuse the 60 frozen E08 tasks but create distinct identities under edit-interface IDs P001--P003 and the E09 execution revision. H018 denotes the shared oracle-file evidence condition; it is not multiplied as a retrieval treatment.

The runner must produce, per cell, run_manifest.json, append-only trajectory.jsonl, every raw model response, messages.json, canonical model.patch, validation.json, and final_metrics.json. Final metrics include attempt/acceptance counts by edit tool, accepted-edit cell status, applicable patch, exact modified files, tests, usage, model residency transitions, and all endpoint stages. Progress and final reports retain every row and counterbalanced order.

Before inference, scripts/preflight_study3.py must validate the clean committed revision, exact 540-cell plan, task/repository pins, all three model variants and 65,536-token loads, tokenizer hashes, one-model residency, tool calling, and deterministic local executor conformance for all three edit interfaces. It must unload all models and stop the server on completion.