| --- |
| license: mit |
| --- |
| |
| # SWE-Bench Trajectory Eval Bundle (v1) |
|
|
| Companion artifact for the trajectory-probe downstream eval of the |
| code-graph-v7 encoders (W1, I6, ...). |
|
|
| ## Contents |
|
|
| - `traj_full_bundle.tar.gz` (488 MB) — contains: |
| - `specs.jsonl`: 2456 SWE-Bench Verified agent trajectories harvested |
| from `swe-bench-submissions` S3 bucket. Fields: instance_id, traj_id, |
| repo, base_commit, patches (1 entry = final model patch), resolved. |
| - `repos/`: shallow (`--filter=blob:none`) clones of the 12 target |
| repos (django, sympy, sphinx, matplotlib, scikit-learn, astropy, |
| xarray, pytest, pylint, requests, seaborn, flask). ~671 MB |
| uncompressed. Blobs pulled lazily per base_commit checkout. |
| - `graphjepa/`: pipeline code (trajectory_pipeline, trajectory_realize, |
| trajectory_probe, trajectory_harvest) plus scripts/trajectory_full.sh. |
| - `harvest.log` — stdout from the S3 harvester that produced specs.jsonl. |
| |
| ## Downstream workflow |
|
|
| ```bash |
| tar -xzf traj_full_bundle.tar.gz |
| rsync -a traj_full/graphjepa/ graphjepa/ |
| mkdir -p outputs/traj_real |
| cp traj_full/specs.jsonl outputs/traj_real/ |
| mv traj_full/repos outputs/traj_real/repos |
| |
| # realize (4 sharded workers by repo) |
| SHARDS=4 bash graphjepa/scripts/trajectory_full.sh |
| tail -f outputs/traj_real/logs/realize_shard*.log |
| |
| # merge manifests + probe with each encoder |
| cat outputs/traj_real/manifest_shard*.jsonl > outputs/traj_real/manifest.jsonl |
| for NAME in W1_softplus_s0 I6_joint_s0; do |
| .venv/bin/python -m graphjepa.trajectory_probe \ |
| --manifest outputs/traj_real/manifest.jsonl \ |
| --ckpt outputs/$NAME/ckpt_final.pt \ |
| --pool mean --split-by repo \ |
| --output outputs/traj_real/probe_${NAME}.json |
| done |
| ``` |
|
|
| ## Provenance |
|
|
| Specs harvested from 5 SWE-Bench Verified submissions: |
|
|
| | Submission | N | Resolved | Rate | |
| |---|---|---|---| |
| | 20240620_sweagent_claude3.5sonnet | 485 | 168 | 34.6% | |
| | 20241022_tools_claude-3-5-sonnet-updated | 483 | 245 | 50.7% | |
| | 20241028_agentless-1.5_gpt4o | 495 | 194 | 39.2% | |
| | 20241029_OpenHands-CodeAct-2.1-sonnet | 493 | 265 | 53.8% | |
| | 20250405_amazon-q-developer-2025 | 500 | 330 | 66.0% | |
| | **total** | **2456** | **1202** | **48.9%** | |
|
|
| 500 unique instance_ids, 499 unique base_commits (median 5 trajectories |
| per commit — different agents attempting the same task). |
|
|
