README.md

---
license: other
task_categories:
  - robotics
tags:
  - robotics
  - teleoperation
  - lerobot
  - vla
  - world-model
  - imitation-learning
  - so-101
pretty_name: DecisionFacts Physical AI Dataset
---

# DecisionFacts Physical AI Dataset — SO-101 Robotic Arm Teleoperation

## Data Summary

This dataset is a curated collection of real-world **teleoperation data** captured on the **SO-101 robotic arm** (`so_follower`), built to support training and evaluation of modern robot-learning models — from imitation-learning policies to large-scale Vision-Language-Action (VLA) and world models.

Each episode is a human-teleoperated demonstration of a manipulation task, recorded synchronously across two camera viewpoints alongside the arm's full proprioceptive state and control signals. Data collection followed a rigorous, standardized protocol:

- **Expert teleoperation** — every episode is a deliberate, goal-directed demonstration performed by a trained operator, not scripted or simulated motion.
- **Synchronized multi-modal capture** — joint states, actions, and dual-camera RGB video are captured in lock-step at 30 fps.
- **Consistency checks** — episodes are reviewed for completeness, timing alignment, and successful task completion before being included in the released dataset.
- **Structured, reproducible recording** — all data is captured and stored using the standardized [LeRobotDataset v3.0](https://huggingface.co/docs/lerobot/main/en/lerobot-dataset-v3) format, ensuring the dataset is immediately compatible with the broader open robot-learning ecosystem.

This combination of careful human demonstration and disciplined data engineering is intended to make the dataset a dependable foundation for downstream policy and representation learning, rather than a loosely collected video corpus.

## Dataset Structure

The dataset follows the **Hugging Face LeRobotDataset v3.0** format. Unlike the earlier v2.1 format (one file per episode), v3.0 packs many episodes into a smaller number of larger, chunked files, with episode boundaries resolved through relational metadata rather than filenames. This makes the dataset scalable, faster to load, and streaming-ready directly from the Hub.

```
physical-ai/
└── SO-101/
    └── <task_name>/                 # e.g. cup_nesting
        ├── data/
        │   └── chunk-000/
        │       └── file-000.parquet         # joint states, actions, indices — many episodes per file
        ├── meta/
        │   ├── info.json                    # schema, fps, robot type, chunking config
        │   ├── stats.json                   # per-feature normalization statistics
        │   ├── tasks.parquet                # task_index -> natural-language task description
        │   └── episodes/
        │       └── chunk-000/
        │           └── file-000.parquet     # per-episode lengths, task refs, file/byte offsets
        └── videos/
            ├── observation.images.cam_front/
            │   └── chunk-000/
            │       └── file-000.mp4          # front-view camera, many episodes per file
            └── observation.images.cam_top/
                └── chunk-000/
                    └── file-000.mp4          # top-down camera, many episodes per file
```

**Key structural points:**

- **`data/`** — Apache Parquet shards containing frame-level `observation.state`, `action` (both 6-DoF: `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_roll`, `gripper`), plus `timestamp`, `frame_index`, `episode_index`, and `task_index`.
- **`videos/`** — Two synchronized camera streams per episode:
  - `observation.images.cam_front` — front-facing view of the workspace
  - `observation.images.cam_top` — top-down view of the workspace
  Both are AV1-encoded MP4 at 480×640, 30 fps, with no audio.
- **`meta/`** — Self-describing metadata: schema/config (`info.json`), normalization stats (`stats.json`), the task-language mapping (`tasks.parquet`), and per-episode index records (`episodes/`).
- **Episodes are not stored as individual folders or files.** Multiple episodes are concatenated into shared, size-capped chunk files (`chunk-000`, `chunk-001`, …). The exact location of any given episode — which chunk, which file, and its frame offset — is resolved by looking it up in `meta/episodes/`, not by filename. This is what allows the dataset to scale to many episodes and tasks without file-system overhead.
- Each task (e.g. `cup_nesting`) is organized as its own self-contained LeRobotDataset directory under `SO-101/`, with its own `data/`, `meta/`, and `videos/` subfolders.

## Methodology

Data collection was designed to produce demonstrations that are diverse and robust enough for policies trained on them to generalize beyond the exact conditions seen during capture:

- **Domain randomization** — object poses, positions, and scene conditions (e.g. object placement, orientation, and workspace configuration) were varied across episodes for each task, rather than repeating a single fixed setup. This exposes downstream models to a broader distribution of visual and spatial conditions during training, reducing overfitting to a narrow demonstration pattern.
- **Episode volume** — a target of **30+ episodes per task** was used to ensure sufficient coverage of the randomized conditions and enough demonstration diversity for stable policy learning.
- **Standardized capture pipeline** — every task follows the same recording protocol: 30 fps synchronized dual-camera capture, 6-DoF joint state/action logging, and consistent episode-level task-language annotation, so tasks are directly comparable and combinable during training.
- **Human-in-the-loop quality control** — demonstrations are performed and reviewed by trained operators to ensure each episode reflects a coherent, successful execution of the intended task.

## Intended Users

This dataset is designed to support several stages of the physical-AI model development stack:

- **VLA (Vision-Language-Action) training** — end-to-end training of policies that map visual observations and language instructions directly to robot actions.
- **Post-training for robotics** — fine-tuning or adapting pretrained robot policies/foundation models to new manipulation tasks using targeted, high-quality demonstration data.
- **VLM pre-training / post-training** — using the paired vision, language (task descriptions), and interaction data to improve grounding of vision-language models in physical, embodied contexts.
- **World model training** — learning predictive models of environment and object dynamics from synchronized multi-view video and action/state sequences.

Researchers and engineers working on imitation learning, robot foundation models, and embodied AI more broadly are the primary intended audience.

## Citation

If you use this dataset in your work, please cite it as follows:

```bibtex
@misc{decisionfacts_teleops_dataset,
  credits      = {Prabhu Raghav, Sreeram B Unni, Balamurugan Pandi, Sriram Gopalan},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/DecisionFacts/physical-ai}}
}
```