DecisionFacts Physical AI Dataset — SO-101 Robotic Arm Teleoperation

Data Summary

This dataset is a curated collection of real-world teleoperation data captured on the SO-101 robotic arm (so_follower), built to support training and evaluation of modern robot-learning models — from imitation-learning policies to large-scale Vision-Language-Action (VLA) and world models.

Each episode is a human-teleoperated demonstration of a manipulation task, recorded synchronously across two camera viewpoints alongside the arm's full proprioceptive state and control signals. Data collection followed a rigorous, standardized protocol:

• Expert teleoperation — every episode is a deliberate, goal-directed demonstration performed by a trained operator, not scripted or simulated motion.
• Synchronized multi-modal capture — joint states, actions, and dual-camera RGB video are captured in lock-step at 30 fps.
• Consistency checks — episodes are reviewed for completeness, timing alignment, and successful task completion before being included in the released dataset.
• Structured, reproducible recording — all data is captured and stored using the standardized LeRobotDataset v3.0 format, ensuring the dataset is immediately compatible with the broader open robot-learning ecosystem.

This combination of careful human demonstration and disciplined data engineering is intended to make the dataset a dependable foundation for downstream policy and representation learning, rather than a loosely collected video corpus.

Dataset Structure

The dataset follows the Hugging Face LeRobotDataset v3.0 format. Unlike the earlier v2.1 format (one file per episode), v3.0 packs many episodes into a smaller number of larger, chunked files, with episode boundaries resolved through relational metadata rather than filenames. This makes the dataset scalable, faster to load, and streaming-ready directly from the Hub.

physical-ai/
└── SO-101/
    └── <task_name>/                 # e.g. cup_nesting
        ├── data/
        │   └── chunk-000/
        │       └── file-000.parquet         # joint states, actions, indices — many episodes per file
        ├── meta/
        │   ├── info.json                    # schema, fps, robot type, chunking config
        │   ├── stats.json                   # per-feature normalization statistics
        │   ├── tasks.parquet                # task_index -> natural-language task description
        │   └── episodes/
        │       └── chunk-000/
        │           └── file-000.parquet     # per-episode lengths, task refs, file/byte offsets
        └── videos/
            ├── observation.images.cam_front/
            │   └── chunk-000/
            │       └── file-000.mp4          # front-view camera, many episodes per file
            └── observation.images.cam_top/
                └── chunk-000/
                    └── file-000.mp4          # top-down camera, many episodes per file

Key structural points:

• data/ — Apache Parquet shards containing frame-level observation.state, action (both 6-DoF: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper), plus timestamp, frame_index, episode_index, and task_index.
• videos/ — Two synchronized camera streams per episode:
  • observation.images.cam_front — front-facing view of the workspace
  • observation.images.cam_top — top-down view of the workspace Both are AV1-encoded MP4 at 480×640, 30 fps, with no audio.
• meta/ — Self-describing metadata: schema/config (info.json), normalization stats (stats.json), the task-language mapping (tasks.parquet), and per-episode index records (episodes/).
• Episodes are not stored as individual folders or files. Multiple episodes are concatenated into shared, size-capped chunk files (chunk-000, chunk-001, …). The exact location of any given episode — which chunk, which file, and its frame offset — is resolved by looking it up in meta/episodes/, not by filename. This is what allows the dataset to scale to many episodes and tasks without file-system overhead.
• Each task (e.g. cup_nesting) is organized as its own self-contained LeRobotDataset directory under SO-101/, with its own data/, meta/, and videos/ subfolders.

Methodology

Data collection was designed to produce demonstrations that are diverse and robust enough for policies trained on them to generalize beyond the exact conditions seen during capture:

• Domain randomization — object poses, positions, and scene conditions (e.g. object placement, orientation, and workspace configuration) were varied across episodes for each task, rather than repeating a single fixed setup. This exposes downstream models to a broader distribution of visual and spatial conditions during training, reducing overfitting to a narrow demonstration pattern.
• Episode volume — a target of 30+ episodes per task was used to ensure sufficient coverage of the randomized conditions and enough demonstration diversity for stable policy learning.
• Standardized capture pipeline — every task follows the same recording protocol: 30 fps synchronized dual-camera capture, 6-DoF joint state/action logging, and consistent episode-level task-language annotation, so tasks are directly comparable and combinable during training.
• Human-in-the-loop quality control — demonstrations are performed and reviewed by trained operators to ensure each episode reflects a coherent, successful execution of the intended task.

Intended Users

This dataset is designed to support several stages of the physical-AI model development stack:

• VLA (Vision-Language-Action) training — end-to-end training of policies that map visual observations and language instructions directly to robot actions.
• Post-training for robotics — fine-tuning or adapting pretrained robot policies/foundation models to new manipulation tasks using targeted, high-quality demonstration data.
• VLM pre-training / post-training — using the paired vision, language (task descriptions), and interaction data to improve grounding of vision-language models in physical, embodied contexts.
• World model training — learning predictive models of environment and object dynamics from synchronized multi-view video and action/state sequences.

Researchers and engineers working on imitation learning, robot foundation models, and embodied AI more broadly are the primary intended audience.

Citation

If you use this dataset in your work, please cite it as follows:

@misc{decisionfacts_teleops_dataset,
  credits      = {Prabhu Raghav, Sreeram B Unni, Balamurugan Pandi, Sriram Gopalan},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/DecisionFacts/physical-ai}}
}