eval_job: add shard mode (--fixtures/--shard-id) for sharded submission eval
Michael Rabinovich+1: Cursor
b2e9d3a
When --fixtures is set, prune the run dir to that shard's slice, evaluate
only it, and upload the per-fixture dirs to reports/<id>/shards/<shard_id>/
for the Space to merge. No --fixtures keeps the original whole-submission
path (report.json + html + gallery) unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
Picks up the report input-render resolver fix (679abc1, resolve inputs
via data_inputs_dir for the two-repo layout) + the #fixture deep-link
hook (1a5766d), so Job-generated reports show editing-task input
renders correctly.
Co-authored-by: Cursor <cursoragent@cursor.com>
upload per-fixture iso render for the leaderboard gallery
Michael Rabinovich+1: Cursor
e5ffda5
After uploading the report bundle, also push renders/<id>/<fixture>.png
(the iso view) per fixture so the gallery has standalone, addressable
thumbnails. Skips fixtures with no iso render (invalid/missing), which
the gallery draws as the dashed cell. The full multi-view renders stay
base64-embedded in reports/<id>.html for the self-contained report.
Co-authored-by: Cursor <cursoragent@cursor.com>
dockerfile: pin cadgenbench to f90bbc5 (render-pool hang fix); deterministic rebuild
Michael Rabinovich+1: Cursor
eb7cf51
Co-authored-by: Cursor <cursoragent@cursor.com>
dockerfile: build cadgenbench from main with commit-endpoint cache-bust
Michael Rabinovich+1: Cursor
8e88c32
Track @main pre-v1 and re-fetch the GitHub commits endpoint so BuildKit
invalidates the pip layer whenever main moves (no more first-build pin).
Co-authored-by: Cursor <cursoragent@cursor.com>
