hydang's picture
|
download
raw
5.15 kB
---
license: mit
task_categories:
- visual-question-answering
- question-answering
language:
- en
tags:
- robotics
- 6dof-pose
- grasping
- spatial-reasoning
- trajectory
- depth-estimation
- benchmark
- evaluation
size_categories:
- n<1K
pretty_name: BOPASK-Test
configs:
- config_name: core-handal
data_files:
- split: test
path: core/bopask-test-handal.json
- config_name: core-hope
data_files:
- split: test
path: core/bopask-test-hope.json
- config_name: core-ycbv
data_files:
- split: test
path: core/bopask-test-ycbv.json
- config_name: lab-home
data_files:
- split: test
path: lab/bopask-test-home.json
---
# BOPASK-Test
Human-verified evaluation benchmark for the **BOPASK** spatial-reasoning VQA dataset.
Contains **934 question-answer pairs** across **two testsets**:
- **`core`** — BOPASK-Core: three BOP-Challenge families (HANDAL, HOPE, YCB-V).
- **`lab`** — BOPASK-Lab : an in-the-wild set of "home / lab" scenes.
## Contents at a glance
| Split | Family | Records | RGB images | Depth maps | Masks |
|-------|---------|--------:|-----------:|-----------:|------:|
| core | handal | 251 | 43 | 41 | 138 |
| core | hope | 189 | 50 | 29 | 231 |
| core | ycbv | 248 | 48 | 48 | 153 |
| lab | home | 246 | 21 | 12 (⚠) | 52 |
| **Total** | | **934** | **162** | **130** | **574** |
## Question-type distribution
| question_type / subtype | handal | hope | ycbv | home | **Total** |
|---|---:|---:|---:|---:|---:|
| pose / 2dbbox | 39 | 39 | 38 | 39 | **155** |
| grasp / 2dplane | 40 | 40 | 40 | 38 | **158** |
| spatial_reasoning / relative_position | 40 | 40 | 40 | 71 | **191** |
| trajectory / 2d | 40 | 40 | 40 | 48 | **168** |
| depth_relative / closer | 40 | — | 40 | 16 | **96** |
| depth_relative / farther | 40 | — | 40 | 24 | **104** |
| object_rearrangement / point_wise | 12 | 30 | 10 | 10 | **62** |
| **family total** | **251** | **189** | **248** | **246** | **934** |
## Layout
```
bopask-test/
├── README.md
├── core/ (BOPASK-Core testset)
│ ├── bopask-test-handal.json
│ ├── bopask-test-hope.json
│ ├── bopask-test-ycbv.json
│ ├── handal/
│ │ ├── images/ (43 *.png)
│ │ ├── depth_maps/ (41 *_depth.png)
│ │ └── masks/ (138 *_mask.png)
│ ├── hope/
│ │ └── images/ depth_maps/ masks/
│ └── ycbv/
│ └── images/ depth_maps/ masks/
└── lab/ (BOPASK-Lab testset)
├── bopask-test-home.json
└── home/
├── images/ (21 *.png)
├── depth_maps/ (empty — see caveat above)
└── masks/ (52 masks_<scene>_<object>.png)
```
All paths inside each JSON are **relative to this dataset root**, e.g.
`core/handal/images/scene_000008_frame_000980.png`.
## Quick start
```python
import json
from datasets import load_dataset
# Load one of the configs:
ds = load_dataset("bhatvineet/bopask-test", "core-handal", split="test")
print(ds[0])
# Or load all four families manually:
configs = ["core-handal", "core-hope", "core-ycbv", "lab-home"]
for cfg in configs:
d = load_dataset("bhatvineet/bopask-test", cfg, split="test")
print(cfg, len(d))
```
Loading directly without `datasets`:
```python
import json
with open("core/bopask-test-handal.json") as f:
records = json.load(f)
for r in records:
img_path = r["images"][0] # e.g. "core/handal/images/..."
user_q = r["messages"][0]["content"]
gt_answer = r["messages"][1]["content"]
```
## Evaluation protocols
Each record is a single-turn VQA pair with one ground-truth response in
`messages[1].content`. Answer formats are self-describing — the user prompt
tells the model the expected output format (e.g. "respond as a list of 2D
points…"). Common metrics by type:
| question_type | typical metric |
|---|---|
| pose / 2dbbox | 2D IoU |
| grasp / 2dplane | endpoint L2 / success@τ |
| trajectory / 2d | trajectory-wise DTW, endpoint error |
| spatial_reasoning / relative_position | exact match (yes/no) |
| depth_relative | exact match (closer/farther) |
| object_rearrangement / point_wise | point-in-mask accuracy |
## Relationship to the training set
This benchmark was curated and human-verified to be disjoint from the
[`bhatvineet/bopask-train`](https://huggingface.co/datasets/bhatvineet/bopask-train)
training split. Use this for evaluation only.
## Citation
If you use this dataset, please cite the [BOPASK](https://arxiv.org/abs/2511.16857) paper
and the underlying BOP-Challenge object-pose datasets
(HANDAL, HOPE, LineMOD, YCB-V).
## License
MIT for the QA annotations. The underlying RGB / depth / mask assets inherit
the licenses of their source BOP-Challenge datasets (HANDAL, HOPE, YCB-V) and
the bopask-home captures.

Xet Storage Details

Size:
5.15 kB
·
Xet hash:
7dd257c89a25acd3faac3cf14c58de0389955077bc8fe9082b607c1e842589bb

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.