codex-benchmark / DATA_FORMAT.md
yzzhao's picture
Publish Codex Benchmark evaluation evidence
0c38e08 verified
|
Raw History Blame Contribute Delete
1.13 kB
# Data format
`data.json` contains the task catalog, per-family summary and completed episode records. `episodes.json` and `episodes.csv` contain the same measured results. Pending tasks have no invented verdict or usage. `attempt-history.json` indexes preserved provider interruptions separately.
Each `episodes/FAMILY/SLOT/seed-0/` directory contains the native verdict, protocol, owner journals, full Codex session, structured trajectory, provider usage records, transcript, observed images, saved resources, video and measured provenance.
Codex native sessions use response_item records. `provider-usage.jsonl` contains native token_usage_record events, not reconstructed HTTP requests. Input includes cached input; output includes reasoning. Failed requests without reported usage add no tokens. Missing usage remains unknown.
Public copies redact account identifiers and local host paths; raw evidence remains unchanged. Workspace archives exclude bytecode caches and use neutral ownership. The manifests describe public bytes. Videos are native spectator recordings edited to 4x speed; they are not simulator replays.