Zhengyiluo commited on
Commit
f4836c4
·
verified ·
1 Parent(s): f23778f

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +90 -78
README.md CHANGED
@@ -55,6 +55,94 @@ pipeline_tag: reinforcement-learning
55
  - Injured walking
56
  - Boxing stance
57
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
  ## Quick Start
59
 
60
  📚 **See the [Quick Start Guide](https://zhengyiluo.github.io/GR00T-WholeBodyControl/getting_started/quickstart.html)** for step-by-step instructions on:
@@ -71,85 +159,9 @@ pipeline_tag: reinforcement-learning
71
 
72
  ## Model Checkpoints
73
 
74
- Pre-trained SONIC policy checkpoints are included in the repository under `gear_sonic_deploy/models/`. The models are optimized for:
75
- - Real-time inference on NVIDIA Jetson platforms
76
- - Low-latency control (< 50ms)
77
- - Robust sim-to-real transfer
78
-
79
- ## Architecture
80
-
81
- SONIC uses a transformer-based policy architecture trained with:
82
- - **Training Framework**: Isaac Lab 2.3.0
83
- - **Backend**: PyTorch
84
- - **Deployment**: C++ inference with TorchScript
85
- - **Control Frequency**: 20-50 Hz
86
- - **Observation Space**: Proprioceptive sensing + motion tracking targets
87
- - **Action Space**: Joint position/velocity targets
88
-
89
- ## Training Data
90
-
91
- SONIC is trained on large-scale human motion capture datasets including:
92
- - Locomotion sequences (walking, running, jumping)
93
- - Full-body manipulation tasks
94
- - Multi-modal movement styles
95
- - Ground-up recovery behaviors
96
-
97
- The model learns through motion tracking, which provides a scalable supervisory signal for acquiring diverse motor skills.
98
-
99
- ## VR Teleoperation
100
-
101
- SONIC supports real-time whole-body teleoperation for data collection and interactive control:
102
-
103
- - **Hardware**: PICO VR headset + controllers
104
- - **Tracking**: Full-body motion retargeting
105
- - **Latency**: < 100ms end-to-end
106
- - **Applications**: Demonstration collection, direct control, behavior annotation
107
-
108
- See the [VR Teleoperation Guide](https://zhengyiluo.github.io/GR00T-WholeBodyControl/tutorials/vr_wholebody_teleop.html) for setup instructions.
109
-
110
- ## Intended Use
111
-
112
- ### Primary Use Cases
113
-
114
- ✅ **Research & Development**
115
- - Humanoid robot control research
116
- - Whole-body motion learning
117
- - Sim-to-real transfer studies
118
- - Teleoperation system development
119
-
120
- ✅ **Robotics Applications**
121
- - Mobile manipulation tasks
122
- - Human-robot interaction
123
- - Demonstration collection
124
- - Behavior cloning baselines
125
-
126
- ### Out-of-Scope Use
127
-
128
- ❌ This model is **NOT** intended for:
129
- - Safety-critical applications without additional validation
130
- - Autonomous operation in uncontrolled environments
131
- - Direct deployment without proper safety measures
132
- - Applications requiring certified safety guarantees
133
-
134
- ## Limitations
135
-
136
- - **Hardware Specific**: Models are tuned for specific humanoid platforms (primarily Unitree G1)
137
- - **Sim-to-Real Gap**: Performance may vary across different hardware configurations
138
- - **Safety**: Requires proper safety protocols during deployment
139
- - **Generalization**: Best performance on behaviors similar to training distribution
140
- - **Environment**: Designed for structured indoor environments
141
-
142
- ## Hardware Requirements
143
-
144
- ### Deployment Platform
145
- - **Recommended**: NVIDIA Jetson Orin (32GB or 64GB)
146
- - **Minimum**: NVIDIA GPU with CUDA support
147
- - **Robot**: Compatible humanoid platforms (Unitree G1, H1, etc.)
148
 
149
- ### Development Platform
150
- - **GPU**: NVIDIA GPU with 16GB+ VRAM (for training)
151
- - **RAM**: 32GB+ recommended
152
- - **Storage**: 100GB+ for datasets and checkpoints
153
 
154
  ## Documentation
155
 
 
55
  - Injured walking
56
  - Boxing stance
57
 
58
+ ## VR Whole-Body Teleoperation
59
+
60
+ SONIC supports real-time whole-body teleoperation via PICO VR headset, enabling natural human-to-robot motion transfer for data collection and interactive control.
61
+
62
+ <div align="center">
63
+ <table>
64
+ <tr>
65
+ <td align="center"><b>Walking</b></td>
66
+ <td align="center"><b>Running</b></td>
67
+ </tr>
68
+ <tr>
69
+ <td align="center"><img src="media/teleop_walking.gif" width="400"></td>
70
+ <td align="center"><img src="media/teleop_running.gif" width="400"></td>
71
+ </tr>
72
+ <tr>
73
+ <td align="center"><b>Sideways Movement</b></td>
74
+ <td align="center"><b>Kneeling</b></td>
75
+ </tr>
76
+ <tr>
77
+ <td align="center"><img src="media/teleop_sideways.gif" width="400"></td>
78
+ <td align="center"><img src="media/teleop_kneeling.gif" width="400"></td>
79
+ </tr>
80
+ <tr>
81
+ <td align="center"><b>Getting Up</b></td>
82
+ <td align="center"><b>Jumping</b></td>
83
+ </tr>
84
+ <tr>
85
+ <td align="center"><img src="media/teleop_getup.gif" width="400"></td>
86
+ <td align="center"><img src="media/teleop_jumping.gif" width="400"></td>
87
+ </tr>
88
+ <tr>
89
+ <td align="center"><b>Bimanual Manipulation</b></td>
90
+ <td align="center"><b>Object Hand-off</b></td>
91
+ </tr>
92
+ <tr>
93
+ <td align="center"><img src="media/teleop_bimanual.gif" width="400"></td>
94
+ <td align="center"><img src="media/teleop_switch_hands.gif" width="400"></td>
95
+ </tr>
96
+ </table>
97
+ </div>
98
+
99
+ ## Kinematic Planner
100
+
101
+ SONIC includes a kinematic planner for real-time locomotion generation — choose a movement style, steer with keyboard/gamepad, and adjust speed and height on the fly.
102
+
103
+ <div align="center">
104
+ <table>
105
+ <tr>
106
+ <td align="center" colspan="2"><b>In-the-Wild Navigation</b></td>
107
+ </tr>
108
+ <tr>
109
+ <td align="center" colspan="2"><img src="media/planner/planner_in_the_wild_navigation.gif" width="800"></td>
110
+ </tr>
111
+ <tr>
112
+ <td align="center"><b>Run</b></td>
113
+ <td align="center"><b>Happy</b></td>
114
+ </tr>
115
+ <tr>
116
+ <td align="center"><img src="media/planner/planner_run.gif" width="400"></td>
117
+ <td align="center"><img src="media/planner/planner_happy.gif" width="400"></td>
118
+ </tr>
119
+ <tr>
120
+ <td align="center"><b>Stealth</b></td>
121
+ <td align="center"><b>Injured</b></td>
122
+ </tr>
123
+ <tr>
124
+ <td align="center"><img src="media/planner/planner_stealth.gif" width="400"></td>
125
+ <td align="center"><img src="media/planner/planner_injured.gif" width="400"></td>
126
+ </tr>
127
+ <tr>
128
+ <td align="center"><b>Kneeling</b></td>
129
+ <td align="center"><b>Hand Crawling</b></td>
130
+ </tr>
131
+ <tr>
132
+ <td align="center"><img src="media/planner/planner_kneeling.gif" width="400"></td>
133
+ <td align="center"><img src="media/planner/planner_hand_crawling.gif" width="400"></td>
134
+ </tr>
135
+ <tr>
136
+ <td align="center"><b>Elbow Crawling</b></td>
137
+ <td align="center"><b>Boxing</b></td>
138
+ </tr>
139
+ <tr>
140
+ <td align="center"><img src="media/planner/planner_elbow_crawling.gif" width="400"></td>
141
+ <td align="center"><img src="media/planner/planner_boxing.gif" width="400"></td>
142
+ </tr>
143
+ </table>
144
+ </div>
145
+
146
  ## Quick Start
147
 
148
  📚 **See the [Quick Start Guide](https://zhengyiluo.github.io/GR00T-WholeBodyControl/getting_started/quickstart.html)** for step-by-step instructions on:
 
159
 
160
  ## Model Checkpoints
161
 
162
+ Pre-trained SONIC policy checkpoints (ONNX format) are available in the `policy/release/` directory of this repository. The models are optimized for real-time inference on NVIDIA Jetson platforms with low-latency control and robust sim-to-real transfer.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
163
 
164
+ See the [Installation Guide](https://zhengyiluo.github.io/GR00T-WholeBodyControl/getting_started/installation_deploy.html) for instructions on downloading and using the model checkpoints.
 
 
 
165
 
166
  ## Documentation
167