PrabhuRaghavDF commited on
Commit
3bca575
Β·
verified Β·
1 Parent(s): bea4155

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +96 -7
README.md CHANGED
@@ -1,10 +1,99 @@
1
  ---
2
- title: README
3
- emoji: πŸš€
4
- colorFrom: red
5
- colorTo: pink
6
- sdk: static
7
- pinned: false
 
 
 
 
 
 
8
  ---
9
 
10
- Edit this `README.md` markdown file to author your organization card.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: other
3
+ task_categories:
4
+ - robotics
5
+ tags:
6
+ - robotics
7
+ - teleoperation
8
+ - lerobot
9
+ - vla
10
+ - world-model
11
+ - imitation-learning
12
+ - so-101
13
+ pretty_name: DecisionFacts Physical AI Dataset
14
  ---
15
 
16
+ # DecisionFacts Physical AI Dataset β€” SO-101 Robotic Arm Teleoperation
17
+
18
+ ## Data Summary
19
+
20
+ This dataset is a curated collection of real-world **teleoperation data** captured on the **SO-101 robotic arm** (`so_follower`), built to support training and evaluation of modern robot-learning models β€” from imitation-learning policies to large-scale Vision-Language-Action (VLA) and world models.
21
+
22
+ Each episode is a human-teleoperated demonstration of a manipulation task, recorded synchronously across two camera viewpoints alongside the arm's full proprioceptive state and control signals. Data collection followed a rigorous, standardized protocol:
23
+
24
+ - **Expert teleoperation** β€” every episode is a deliberate, goal-directed demonstration performed by a trained operator, not scripted or simulated motion.
25
+ - **Synchronized multi-modal capture** β€” joint states, actions, and dual-camera RGB video are captured in lock-step at 30 fps.
26
+ - **Consistency checks** β€” episodes are reviewed for completeness, timing alignment, and successful task completion before being included in the released dataset.
27
+ - **Structured, reproducible recording** β€” all data is captured and stored using the standardized [LeRobotDataset v3.0](https://huggingface.co/docs/lerobot/main/en/lerobot-dataset-v3) format, ensuring the dataset is immediately compatible with the broader open robot-learning ecosystem.
28
+
29
+ This combination of careful human demonstration and disciplined data engineering is intended to make the dataset a dependable foundation for downstream policy and representation learning, rather than a loosely collected video corpus.
30
+
31
+ ## Dataset Structure
32
+
33
+ The dataset follows the **Hugging Face LeRobotDataset v3.0** format. Unlike the earlier v2.1 format (one file per episode), v3.0 packs many episodes into a smaller number of larger, chunked files, with episode boundaries resolved through relational metadata rather than filenames. This makes the dataset scalable, faster to load, and streaming-ready directly from the Hub.
34
+
35
+ ```
36
+ physical-ai/
37
+ └── SO-101/
38
+ └── <task_name>/ # e.g. cup_nesting
39
+ β”œβ”€β”€ data/
40
+ β”‚ └── chunk-000/
41
+ β”‚ └── file-000.parquet # joint states, actions, indices β€” many episodes per file
42
+ β”œβ”€β”€ meta/
43
+ β”‚ β”œβ”€β”€ info.json # schema, fps, robot type, chunking config
44
+ β”‚ β”œβ”€β”€ stats.json # per-feature normalization statistics
45
+ β”‚ β”œβ”€β”€ tasks.parquet # task_index -> natural-language task description
46
+ β”‚ └── episodes/
47
+ β”‚ └── chunk-000/
48
+ β”‚ └── file-000.parquet # per-episode lengths, task refs, file/byte offsets
49
+ └── videos/
50
+ β”œβ”€β”€ observation.images.cam_front/
51
+ β”‚ └── chunk-000/
52
+ β”‚ └── file-000.mp4 # front-view camera, many episodes per file
53
+ └── observation.images.cam_top/
54
+ └── chunk-000/
55
+ └── file-000.mp4 # top-down camera, many episodes per file
56
+ ```
57
+
58
+ **Key structural points:**
59
+
60
+ - **`data/`** β€” Apache Parquet shards containing frame-level `observation.state`, `action` (both 6-DoF: `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_roll`, `gripper`), plus `timestamp`, `frame_index`, `episode_index`, and `task_index`.
61
+ - **`videos/`** β€” Two synchronized camera streams per episode:
62
+ - `observation.images.cam_front` β€” front-facing view of the workspace
63
+ - `observation.images.cam_top` β€” top-down view of the workspace
64
+ Both are AV1-encoded MP4 at 480Γ—640, 30 fps, with no audio.
65
+ - **`meta/`** β€” Self-describing metadata: schema/config (`info.json`), normalization stats (`stats.json`), the task-language mapping (`tasks.parquet`), and per-episode index records (`episodes/`).
66
+ - **Episodes are not stored as individual folders or files.** Multiple episodes are concatenated into shared, size-capped chunk files (`chunk-000`, `chunk-001`, …). The exact location of any given episode β€” which chunk, which file, and its frame offset β€” is resolved by looking it up in `meta/episodes/`, not by filename. This is what allows the dataset to scale to many episodes and tasks without file-system overhead.
67
+ - Each task (e.g. `cup_nesting`) is organized as its own self-contained LeRobotDataset directory under `SO-101/`, with its own `data/`, `meta/`, and `videos/` subfolders.
68
+
69
+ ## Methodology
70
+
71
+ Data collection was designed to produce demonstrations that are diverse and robust enough for policies trained on them to generalize beyond the exact conditions seen during capture:
72
+
73
+ - **Domain randomization** β€” object poses, positions, and scene conditions (e.g. object placement, orientation, and workspace configuration) were varied across episodes for each task, rather than repeating a single fixed setup. This exposes downstream models to a broader distribution of visual and spatial conditions during training, reducing overfitting to a narrow demonstration pattern.
74
+ - **Episode volume** β€” a target of **30+ episodes per task** was used to ensure sufficient coverage of the randomized conditions and enough demonstration diversity for stable policy learning.
75
+ - **Standardized capture pipeline** β€” every task follows the same recording protocol: 30 fps synchronized dual-camera capture, 6-DoF joint state/action logging, and consistent episode-level task-language annotation, so tasks are directly comparable and combinable during training.
76
+ - **Human-in-the-loop quality control** β€” demonstrations are performed and reviewed by trained operators to ensure each episode reflects a coherent, successful execution of the intended task.
77
+
78
+ ## Intended Users
79
+
80
+ This dataset is designed to support several stages of the physical-AI model development stack:
81
+
82
+ - **VLA (Vision-Language-Action) training** β€” end-to-end training of policies that map visual observations and language instructions directly to robot actions.
83
+ - **Post-training for robotics** β€” fine-tuning or adapting pretrained robot policies/foundation models to new manipulation tasks using targeted, high-quality demonstration data.
84
+ - **VLM pre-training / post-training** β€” using the paired vision, language (task descriptions), and interaction data to improve grounding of vision-language models in physical, embodied contexts.
85
+ - **World model training** β€” learning predictive models of environment and object dynamics from synchronized multi-view video and action/state sequences.
86
+
87
+ Researchers and engineers working on imitation learning, robot foundation models, and embodied AI more broadly are the primary intended audience.
88
+
89
+ ## Citation
90
+
91
+ If you use this dataset in your work, please cite it as follows:
92
+
93
+ ```bibtex
94
+ @misc{decisionfacts_teleops_dataset,
95
+ credits = {Prabhu Raghav, Sreeram B Uni, Balamurugan Pandi, Sriram Gopalan},
96
+ year = {2026},
97
+ howpublished = {\url{https://huggingface.co/datasets/DecisionFacts/physical-ai}}
98
+ }
99
+ ```