File size: 3,311 Bytes
53c10a4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | # Evaluation runtime
`eval/task/` holds the evaluators that produce every number in the paper. They
ship with this repository, so a single clone can reproduce the full suite.
`orarl-eval` is the supported entry point: it resolves the canonical dataset
layout into per-task environment variables and then runs `eval/task/eval.sh`.
```bash
orarl-eval \
--model /path/to/exported-model \
--tasks paper \
--dataset OraRL/OraRL-Data \
--summary ./evaluation.json \
--run
```
Without `--run` the command prints the resolved plan and exits, which is the
fastest way to confirm data and model paths before allocating GPUs.
`eval/task/eval.sh` can also be called directly when you want to bypass the
canonical dataset resolution and set the per-task variables yourself.
## Task map
| Family | Benchmarks | Entry point |
| --- | --- | --- |
| Video QA | VideoMME, VideoMME-v2, MVBench, MMVU, Video-Holmes, LongVideoBench, MLVU | `eval_vllm.py` |
| Spatial intelligence | VSI-Bench, MindCube, MMSI-Bench, ReVSI | `eval_vllm.py`, `mmsi/eval_mmsi_transformers.py`, `revsi/eval_revsi_vllm.py` |
| Temporal grounding | TimeLens (Charades, ActivityNet, QVHighlights) | `temporal_grounding/eval_timelens_hf.py` |
| Spatial grounding | RefCOCO, RefCOCO+, RefCOCOg | `spatial_grounding/eval_refcoco_vllm.py` |
| Tracking | GOT-10k | `tracking/eval_tracking_vllm.py` |
| Spatio-temporal grounding | STVG | `spatial_temporal_grounding/eval_stvg_vllm.py` |
| Segmentation | RefCOCO series, MeViS, ReasonVOS | `segmentation/eval_seg_vllm.py` plus `segmentation/post_sam2.py` |
`eval_prompt.py` is the single source of truth for prompts, and
`canonical_data.py` adapts the canonical `ORARL_EVAL_*` layout for every
evaluator. MMSI-Bench runs through Transformers rather than vLLM; the other
families run through vLLM.
## Assets you must obtain separately
Annotations and media come from the `OraRL/OraRL-Data` dataset repository, and
model weights come from the released checkpoints. Neither is vendored here.
Segmentation additionally needs three inputs, all supplied by you under their
upstream licenses:
- SAM2 weights and the matching Hydra config, passed as
`SEGMENTATION_SAM2_CKPT` and `SEGMENTATION_SAM2_CFG`.
- The official OneThinker `seg_post_sam2.py`, passed as
`SEGMENTATION_POSTPROCESSOR_PATH`. `segmentation/post_sam2.py` is a thin
wrapper that injects paths into it so mask metrics stay identical to the
upstream implementation.
- The `sam2` Python package.
`orarl-eval` refuses to start segmentation with `--segmentation-run-sam2`
unless all three paths exist.
## Video decoding
Video evaluators pin the decord backend through `FORCE_QWENVL_VIDEO_READER`.
`eval/task/qwenvl_decord_patch.py` replaces `qwen_vl_utils.fetch_video` so the
backend is honoured across `qwen_vl_utils` releases that otherwise hard-code
torchvision. The backend is not cosmetic: switching MeViS and ReasonVOS from
torchcodec to decord moved MeViS J&F from 56.7 to 60.6. Set
`SEGMENTATION_VIDEO_READER` to `torchcodec` or `torchvision` only when
deliberately measuring that difference.
## Checkpoint format
Exported Hugging Face checkpoints run as-is. Pass `--force-merge` when the model
path is a sharded FSDP actor directory; `eval/task/eval.sh` then merges it with
`scripts/model_merger.py` before inference.
|