Spaces:
Configuration error
Configuration error
| license: other | |
| task_categories: | |
| - robotics | |
| tags: | |
| - robotics | |
| - teleoperation | |
| - lerobot | |
| - vla | |
| - world-model | |
| - imitation-learning | |
| - so-101 | |
| pretty_name: DecisionFacts Physical AI Dataset | |
| # DecisionFacts Physical AI Dataset — SO-101 Robotic Arm Teleoperation | |
| ## Data Summary | |
| This dataset is a curated collection of real-world **teleoperation data** captured on the **SO-101 robotic arm** (`so_follower`), built to support training and evaluation of modern robot-learning models — from imitation-learning policies to large-scale Vision-Language-Action (VLA) and world models. | |
| Each episode is a human-teleoperated demonstration of a manipulation task, recorded synchronously across two camera viewpoints alongside the arm's full proprioceptive state and control signals. Data collection followed a rigorous, standardized protocol: | |
| - **Expert teleoperation** — every episode is a deliberate, goal-directed demonstration performed by a trained operator, not scripted or simulated motion. | |
| - **Synchronized multi-modal capture** — joint states, actions, and dual-camera RGB video are captured in lock-step at 30 fps. | |
| - **Consistency checks** — episodes are reviewed for completeness, timing alignment, and successful task completion before being included in the released dataset. | |
| - **Structured, reproducible recording** — all data is captured and stored using the standardized [LeRobotDataset v3.0](https://huggingface.co/docs/lerobot/main/en/lerobot-dataset-v3) format, ensuring the dataset is immediately compatible with the broader open robot-learning ecosystem. | |
| This combination of careful human demonstration and disciplined data engineering is intended to make the dataset a dependable foundation for downstream policy and representation learning, rather than a loosely collected video corpus. | |
| ## Dataset Structure | |
| The dataset follows the **Hugging Face LeRobotDataset v3.0** format. Unlike the earlier v2.1 format (one file per episode), v3.0 packs many episodes into a smaller number of larger, chunked files, with episode boundaries resolved through relational metadata rather than filenames. This makes the dataset scalable, faster to load, and streaming-ready directly from the Hub. | |
| ``` | |
| physical-ai/ | |
| └── SO-101/ | |
| └── <task_name>/ # e.g. cup_nesting | |
| ├── data/ | |
| │ └── chunk-000/ | |
| │ └── file-000.parquet # joint states, actions, indices — many episodes per file | |
| ├── meta/ | |
| │ ├── info.json # schema, fps, robot type, chunking config | |
| │ ├── stats.json # per-feature normalization statistics | |
| │ ├── tasks.parquet # task_index -> natural-language task description | |
| │ └── episodes/ | |
| │ └── chunk-000/ | |
| │ └── file-000.parquet # per-episode lengths, task refs, file/byte offsets | |
| └── videos/ | |
| ├── observation.images.cam_front/ | |
| │ └── chunk-000/ | |
| │ └── file-000.mp4 # front-view camera, many episodes per file | |
| └── observation.images.cam_top/ | |
| └── chunk-000/ | |
| └── file-000.mp4 # top-down camera, many episodes per file | |
| ``` | |
| **Key structural points:** | |
| - **`data/`** — Apache Parquet shards containing frame-level `observation.state`, `action` (both 6-DoF: `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_roll`, `gripper`), plus `timestamp`, `frame_index`, `episode_index`, and `task_index`. | |
| - **`videos/`** — Two synchronized camera streams per episode: | |
| - `observation.images.cam_front` — front-facing view of the workspace | |
| - `observation.images.cam_top` — top-down view of the workspace | |
| Both are AV1-encoded MP4 at 480×640, 30 fps, with no audio. | |
| - **`meta/`** — Self-describing metadata: schema/config (`info.json`), normalization stats (`stats.json`), the task-language mapping (`tasks.parquet`), and per-episode index records (`episodes/`). | |
| - **Episodes are not stored as individual folders or files.** Multiple episodes are concatenated into shared, size-capped chunk files (`chunk-000`, `chunk-001`, …). The exact location of any given episode — which chunk, which file, and its frame offset — is resolved by looking it up in `meta/episodes/`, not by filename. This is what allows the dataset to scale to many episodes and tasks without file-system overhead. | |
| - Each task (e.g. `cup_nesting`) is organized as its own self-contained LeRobotDataset directory under `SO-101/`, with its own `data/`, `meta/`, and `videos/` subfolders. | |
| ## Methodology | |
| Data collection was designed to produce demonstrations that are diverse and robust enough for policies trained on them to generalize beyond the exact conditions seen during capture: | |
| - **Domain randomization** — object poses, positions, and scene conditions (e.g. object placement, orientation, and workspace configuration) were varied across episodes for each task, rather than repeating a single fixed setup. This exposes downstream models to a broader distribution of visual and spatial conditions during training, reducing overfitting to a narrow demonstration pattern. | |
| - **Episode volume** — a target of **30+ episodes per task** was used to ensure sufficient coverage of the randomized conditions and enough demonstration diversity for stable policy learning. | |
| - **Standardized capture pipeline** — every task follows the same recording protocol: 30 fps synchronized dual-camera capture, 6-DoF joint state/action logging, and consistent episode-level task-language annotation, so tasks are directly comparable and combinable during training. | |
| - **Human-in-the-loop quality control** — demonstrations are performed and reviewed by trained operators to ensure each episode reflects a coherent, successful execution of the intended task. | |
| ## Intended Users | |
| This dataset is designed to support several stages of the physical-AI model development stack: | |
| - **VLA (Vision-Language-Action) training** — end-to-end training of policies that map visual observations and language instructions directly to robot actions. | |
| - **Post-training for robotics** — fine-tuning or adapting pretrained robot policies/foundation models to new manipulation tasks using targeted, high-quality demonstration data. | |
| - **VLM pre-training / post-training** — using the paired vision, language (task descriptions), and interaction data to improve grounding of vision-language models in physical, embodied contexts. | |
| - **World model training** — learning predictive models of environment and object dynamics from synchronized multi-view video and action/state sequences. | |
| Researchers and engineers working on imitation learning, robot foundation models, and embodied AI more broadly are the primary intended audience. | |
| ## Citation | |
| If you use this dataset in your work, please cite it as follows: | |
| ```bibtex | |
| @misc{decisionfacts_teleops_dataset, | |
| credits = {Prabhu Raghav, Sreeram B Unni, Balamurugan Pandi, Sriram Gopalan}, | |
| year = {2026}, | |
| howpublished = {\url{https://huggingface.co/datasets/DecisionFacts/physical-ai}} | |
| } | |
| ``` | |