Spaces:
Configuration error
Configuration error
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,10 +1,99 @@
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
---
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: other
|
| 3 |
+
task_categories:
|
| 4 |
+
- robotics
|
| 5 |
+
tags:
|
| 6 |
+
- robotics
|
| 7 |
+
- teleoperation
|
| 8 |
+
- lerobot
|
| 9 |
+
- vla
|
| 10 |
+
- world-model
|
| 11 |
+
- imitation-learning
|
| 12 |
+
- so-101
|
| 13 |
+
pretty_name: DecisionFacts Physical AI Dataset
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# DecisionFacts Physical AI Dataset β SO-101 Robotic Arm Teleoperation
|
| 17 |
+
|
| 18 |
+
## Data Summary
|
| 19 |
+
|
| 20 |
+
This dataset is a curated collection of real-world **teleoperation data** captured on the **SO-101 robotic arm** (`so_follower`), built to support training and evaluation of modern robot-learning models β from imitation-learning policies to large-scale Vision-Language-Action (VLA) and world models.
|
| 21 |
+
|
| 22 |
+
Each episode is a human-teleoperated demonstration of a manipulation task, recorded synchronously across two camera viewpoints alongside the arm's full proprioceptive state and control signals. Data collection followed a rigorous, standardized protocol:
|
| 23 |
+
|
| 24 |
+
- **Expert teleoperation** β every episode is a deliberate, goal-directed demonstration performed by a trained operator, not scripted or simulated motion.
|
| 25 |
+
- **Synchronized multi-modal capture** β joint states, actions, and dual-camera RGB video are captured in lock-step at 30 fps.
|
| 26 |
+
- **Consistency checks** β episodes are reviewed for completeness, timing alignment, and successful task completion before being included in the released dataset.
|
| 27 |
+
- **Structured, reproducible recording** β all data is captured and stored using the standardized [LeRobotDataset v3.0](https://huggingface.co/docs/lerobot/main/en/lerobot-dataset-v3) format, ensuring the dataset is immediately compatible with the broader open robot-learning ecosystem.
|
| 28 |
+
|
| 29 |
+
This combination of careful human demonstration and disciplined data engineering is intended to make the dataset a dependable foundation for downstream policy and representation learning, rather than a loosely collected video corpus.
|
| 30 |
+
|
| 31 |
+
## Dataset Structure
|
| 32 |
+
|
| 33 |
+
The dataset follows the **Hugging Face LeRobotDataset v3.0** format. Unlike the earlier v2.1 format (one file per episode), v3.0 packs many episodes into a smaller number of larger, chunked files, with episode boundaries resolved through relational metadata rather than filenames. This makes the dataset scalable, faster to load, and streaming-ready directly from the Hub.
|
| 34 |
+
|
| 35 |
+
```
|
| 36 |
+
physical-ai/
|
| 37 |
+
βββ SO-101/
|
| 38 |
+
βββ <task_name>/ # e.g. cup_nesting
|
| 39 |
+
βββ data/
|
| 40 |
+
β βββ chunk-000/
|
| 41 |
+
β βββ file-000.parquet # joint states, actions, indices β many episodes per file
|
| 42 |
+
βββ meta/
|
| 43 |
+
β βββ info.json # schema, fps, robot type, chunking config
|
| 44 |
+
β βββ stats.json # per-feature normalization statistics
|
| 45 |
+
β βββ tasks.parquet # task_index -> natural-language task description
|
| 46 |
+
β βββ episodes/
|
| 47 |
+
β βββ chunk-000/
|
| 48 |
+
β βββ file-000.parquet # per-episode lengths, task refs, file/byte offsets
|
| 49 |
+
βββ videos/
|
| 50 |
+
βββ observation.images.cam_front/
|
| 51 |
+
β βββ chunk-000/
|
| 52 |
+
β βββ file-000.mp4 # front-view camera, many episodes per file
|
| 53 |
+
βββ observation.images.cam_top/
|
| 54 |
+
βββ chunk-000/
|
| 55 |
+
βββ file-000.mp4 # top-down camera, many episodes per file
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
**Key structural points:**
|
| 59 |
+
|
| 60 |
+
- **`data/`** β Apache Parquet shards containing frame-level `observation.state`, `action` (both 6-DoF: `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_roll`, `gripper`), plus `timestamp`, `frame_index`, `episode_index`, and `task_index`.
|
| 61 |
+
- **`videos/`** β Two synchronized camera streams per episode:
|
| 62 |
+
- `observation.images.cam_front` β front-facing view of the workspace
|
| 63 |
+
- `observation.images.cam_top` β top-down view of the workspace
|
| 64 |
+
Both are AV1-encoded MP4 at 480Γ640, 30 fps, with no audio.
|
| 65 |
+
- **`meta/`** β Self-describing metadata: schema/config (`info.json`), normalization stats (`stats.json`), the task-language mapping (`tasks.parquet`), and per-episode index records (`episodes/`).
|
| 66 |
+
- **Episodes are not stored as individual folders or files.** Multiple episodes are concatenated into shared, size-capped chunk files (`chunk-000`, `chunk-001`, β¦). The exact location of any given episode β which chunk, which file, and its frame offset β is resolved by looking it up in `meta/episodes/`, not by filename. This is what allows the dataset to scale to many episodes and tasks without file-system overhead.
|
| 67 |
+
- Each task (e.g. `cup_nesting`) is organized as its own self-contained LeRobotDataset directory under `SO-101/`, with its own `data/`, `meta/`, and `videos/` subfolders.
|
| 68 |
+
|
| 69 |
+
## Methodology
|
| 70 |
+
|
| 71 |
+
Data collection was designed to produce demonstrations that are diverse and robust enough for policies trained on them to generalize beyond the exact conditions seen during capture:
|
| 72 |
+
|
| 73 |
+
- **Domain randomization** β object poses, positions, and scene conditions (e.g. object placement, orientation, and workspace configuration) were varied across episodes for each task, rather than repeating a single fixed setup. This exposes downstream models to a broader distribution of visual and spatial conditions during training, reducing overfitting to a narrow demonstration pattern.
|
| 74 |
+
- **Episode volume** β a target of **30+ episodes per task** was used to ensure sufficient coverage of the randomized conditions and enough demonstration diversity for stable policy learning.
|
| 75 |
+
- **Standardized capture pipeline** β every task follows the same recording protocol: 30 fps synchronized dual-camera capture, 6-DoF joint state/action logging, and consistent episode-level task-language annotation, so tasks are directly comparable and combinable during training.
|
| 76 |
+
- **Human-in-the-loop quality control** β demonstrations are performed and reviewed by trained operators to ensure each episode reflects a coherent, successful execution of the intended task.
|
| 77 |
+
|
| 78 |
+
## Intended Users
|
| 79 |
+
|
| 80 |
+
This dataset is designed to support several stages of the physical-AI model development stack:
|
| 81 |
+
|
| 82 |
+
- **VLA (Vision-Language-Action) training** β end-to-end training of policies that map visual observations and language instructions directly to robot actions.
|
| 83 |
+
- **Post-training for robotics** β fine-tuning or adapting pretrained robot policies/foundation models to new manipulation tasks using targeted, high-quality demonstration data.
|
| 84 |
+
- **VLM pre-training / post-training** β using the paired vision, language (task descriptions), and interaction data to improve grounding of vision-language models in physical, embodied contexts.
|
| 85 |
+
- **World model training** β learning predictive models of environment and object dynamics from synchronized multi-view video and action/state sequences.
|
| 86 |
+
|
| 87 |
+
Researchers and engineers working on imitation learning, robot foundation models, and embodied AI more broadly are the primary intended audience.
|
| 88 |
+
|
| 89 |
+
## Citation
|
| 90 |
+
|
| 91 |
+
If you use this dataset in your work, please cite it as follows:
|
| 92 |
+
|
| 93 |
+
```bibtex
|
| 94 |
+
@misc{decisionfacts_teleops_dataset,
|
| 95 |
+
credits = {Prabhu Raghav, Sreeram B Uni, Balamurugan Pandi, Sriram Gopalan},
|
| 96 |
+
year = {2026},
|
| 97 |
+
howpublished = {\url{https://huggingface.co/datasets/DecisionFacts/physical-ai}}
|
| 98 |
+
}
|
| 99 |
+
```
|