File size: 599 Bytes
d61821a | 1 2 3 4 5 6 7 8 9 10 11 12 13 | # Task data
Each benchmark task gets one TOML file under `manifests/`. The example is a
schema template, not an eligible experiment task. Split files contain one
`task_id` per line and must be frozen before confirmatory evaluation.
Gold patches, gold locations, and hidden tests are evaluator-only data. The
agent receives only the problem statement and repository state, except in the
explicit oracle controls. Public repository URLs and exact base commits are
required for reproducibility; obtain and preserve any repository license and
redistribution metadata before publishing a task bundle.
|