README.md

# CUAStepBench leaderboard judge outputs

Raw judge outputs behind every row of the CUAStepBench leaderboard, one folder per
row. CUAStepBench itself (goals, screenshots, actions and the `human-annotated-zzh`
ground truth) comes from
[huggingface.co/datasets/ZJUSCL/CUAStepBench](https://huggingface.co/datasets/ZJUSCL/CUAStepBench).

| Folder | Framework | Model | Runs |
|---|---|---|---:|
| `01-codex-gpt-5.5` | Codex | GPT-5.5 | 1 |
| `02-seekjudge-deepseek-v4-pro` | SeekJudge | DeepSeek-V4-Pro + Gemini-3.0-Flash | 2 |
| `03-cuajudge-gpt-5-mini` | CUAJudge | GPT-5-mini | 1 |
| `04-seekjudge-9b` | SeekJudge | SeekJudge-9B | 4 |
| `05-seekjudge-qwen3vl-8b` | SeekJudge | Qwen3VL-8B | 6 |
| `06-cuajudge-qwen3vl-8b` | CUAJudge | Qwen3VL-8B | 1 |
| `07-osthemis-qwen3vl-8b` | OSThemis | Qwen3VL-8B | 3 |

Each folder holds a `README.md`, a `metrics.json`, a `judge_name_mapping.json`
inventory, and a `CUAStepBench/` tree mirroring the released dataset layout:

```
CUAStepBench/<benchmark>/<agent>/<task_id>/judge/<run>/result.json
```

## Using them

```bash
# merge one row (or all seven) into a CUAStepBench checkout
rsync -a 04-seekjudge-9b/CUAStepBench/ /path/to/CUAStepBench/

# analyse it with the SeekJudge release code
REWARD_SUBCRITERIA_MODEL_PATH=./configs/subcriteria/fitted_kfold_P84_xgb_md2_ft_3bench.pkl \
python -m seekjudge.analysis.analyse_judge \
    --data-dir /path/to/CUAStepBench \
    --gt-judge-name human-annotated-zzh \
    --target-judge-name seekjudge-9b \
    --step-9class2class-config configs/subcriteria_step/9classmap.yaml
```

The leaderboard numbers use the
[`fitted_kfold_P84_xgb_md2_ft_3bench.pkl`](https://github.com/ZJUSCL/SeekJudge/blob/4dd0c5172b09242faf7243b8ce9d6d18808ac3d1/configs/subcriteria/fitted_kfold_P84_xgb_md2_ft_3bench.pkl)
scorer.

The run directories keep the `_rand1` .. `_rand5` suffix convention, which is how
`analyse_judge` finds the repeated samples of one configuration. Passing the base
name picks up every repeat.