README.md

---
size_categories:
- 10K<n<100K
license: mit
---

# Dataset Card for RoboFAC
[![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://mint-sjtu.github.io/RoboFAC.io/)
[![Github](https://img.shields.io/badge/Project-Github-light)](https://github.com/MINT-SJTU/RoboFAC)
[![Paper](https://img.shields.io/badge/Paper-PDF-red)](https://arxiv.org/abs/2505.12224)
[![Dataset](https://img.shields.io/badge/Dataset-Huggingface-green)](https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset)
[![Model](https://img.shields.io/badge/Model-Huggingface-yellow)](https://huggingface.co/MINT-SJTU/RoboFAC-7B)

RoboFAC is a multimodal VQA dataset for robotic failure analysis and correction, comprising over **10,000** robot manipulation videos and **78,623** question-answer pairs. The dataset supports both **VQA** and **failure reasoning** tasks across simulated and real-world environments.

## Files
```
# Make sure git-lfs is installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset
```
## Dataset Details

### Dataset Description

RoboFAC is designed to benchmark multimodal reasoning and error correction in robot manipulation. It includes:

- **9,440** erroneous robot manipulation trajectories
- **78,623** QA pairs
- **16** diverse manipulation tasks
- **53** unique scenes
- **1282** successful robot trajectories (as reference)

The dataset contains **simulated** and **real-world** demonstrations and is organized to support both **training** and **evaluation** of vision-language and robotic reasoning models.

- **Curated by:** MINT Group @ SJTU

- **Language(s) (NLP):** English
- **License:** MIT License

### Tasks and Descriptions

| Task Name         | Description                                                                 |
|-------------------|-----------------------------------------------------------------------------|
| SpinStack         | Pick up the cube on the spinning disc and stack it on another cube on the disc. |
| SpinPullStack     | Pull out the cube on the spinning disc and stack it on another cube on the disc. |
| MicrowaveTask     | Put the spoon on the table into the cup. Open the door of microwave, put the cup into the microwave and close the door. |
| SafeTask          | Put the gold bar into the safe, close the door of the safe and rotate the cross knob on the door to lock it. |
| ToolsTask         | Choose the correct (L-shaped) tools, grasp it to pull the correct (2-pins) charger and plug it. |
| UprightStask      | Upright the peg and stack it on the cube.                                     |
| PegInsetionSide   | Insert the peg into the hole on the side of the block.                        |
| PullCubeTool      | Grasp the L-shaped tool and pull the cube by it.                              |
| PlugCharger       | Grasp the charger and plug it into the receptacle.                            |
| InsertCylinder    | Upright the cylinder and insert it into the middle hole on the shelf.         |
| PlaceCube         | Pick up the cube and place it into the box.                                   |
| LiftPegUpright    | Lift the peg and upright it.                                                  |
| PickCube          | Pick the cube to the target position.                                        |
| PullCube          | Pull the cube to the red and white target.                                   |
| PushCube          | Push the cube to the red and white target.                                   |
| StackCube         | Pick up the cube and stack it on another cube.                               |

### Dataset Sources

- **Repository:** [https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset](https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset)
- **Paper:** [https://arxiv.org/abs/2505.12224](https://arxiv.org/abs/2505.12224)


## Uses


### Direct Use

RoboFAC is intended for:

- Robotic failure detection and correction
- Video-based VQA for robot actions
- Evaluating robotic vision-language models

Access the observation and action data:

* There are a `.h5` file and a `.json` file in each leaf directory
* In the `.json` file, we have `episode_id` and `unique_id` for each trajectories
* The `unique_id` relates to the video name, and you can find corresponding kinematics data in `.h5` file by the `episode_id`

## Dataset Structure

Each item in `training_qa.json` contains:

```json
{
  "id": "uuid",
  "video": "task/scenario/video_id.mp4",
  "conversations": [
    {
      "from": "human",
      "value": "<video>\\n[Prompt Question]"
    },
    {
      "from": "assistant",
      "value": "[Answer from model or annotator]"
    }
  ]
}