Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -55,6 +55,94 @@ pipeline_tag: reinforcement-learning
|
|
| 55 |
- Injured walking
|
| 56 |
- Boxing stance
|
| 57 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
## Quick Start
|
| 59 |
|
| 60 |
📚 **See the [Quick Start Guide](https://zhengyiluo.github.io/GR00T-WholeBodyControl/getting_started/quickstart.html)** for step-by-step instructions on:
|
|
@@ -71,85 +159,9 @@ pipeline_tag: reinforcement-learning
|
|
| 71 |
|
| 72 |
## Model Checkpoints
|
| 73 |
|
| 74 |
-
Pre-trained SONIC policy checkpoints are
|
| 75 |
-
- Real-time inference on NVIDIA Jetson platforms
|
| 76 |
-
- Low-latency control (< 50ms)
|
| 77 |
-
- Robust sim-to-real transfer
|
| 78 |
-
|
| 79 |
-
## Architecture
|
| 80 |
-
|
| 81 |
-
SONIC uses a transformer-based policy architecture trained with:
|
| 82 |
-
- **Training Framework**: Isaac Lab 2.3.0
|
| 83 |
-
- **Backend**: PyTorch
|
| 84 |
-
- **Deployment**: C++ inference with TorchScript
|
| 85 |
-
- **Control Frequency**: 20-50 Hz
|
| 86 |
-
- **Observation Space**: Proprioceptive sensing + motion tracking targets
|
| 87 |
-
- **Action Space**: Joint position/velocity targets
|
| 88 |
-
|
| 89 |
-
## Training Data
|
| 90 |
-
|
| 91 |
-
SONIC is trained on large-scale human motion capture datasets including:
|
| 92 |
-
- Locomotion sequences (walking, running, jumping)
|
| 93 |
-
- Full-body manipulation tasks
|
| 94 |
-
- Multi-modal movement styles
|
| 95 |
-
- Ground-up recovery behaviors
|
| 96 |
-
|
| 97 |
-
The model learns through motion tracking, which provides a scalable supervisory signal for acquiring diverse motor skills.
|
| 98 |
-
|
| 99 |
-
## VR Teleoperation
|
| 100 |
-
|
| 101 |
-
SONIC supports real-time whole-body teleoperation for data collection and interactive control:
|
| 102 |
-
|
| 103 |
-
- **Hardware**: PICO VR headset + controllers
|
| 104 |
-
- **Tracking**: Full-body motion retargeting
|
| 105 |
-
- **Latency**: < 100ms end-to-end
|
| 106 |
-
- **Applications**: Demonstration collection, direct control, behavior annotation
|
| 107 |
-
|
| 108 |
-
See the [VR Teleoperation Guide](https://zhengyiluo.github.io/GR00T-WholeBodyControl/tutorials/vr_wholebody_teleop.html) for setup instructions.
|
| 109 |
-
|
| 110 |
-
## Intended Use
|
| 111 |
-
|
| 112 |
-
### Primary Use Cases
|
| 113 |
-
|
| 114 |
-
✅ **Research & Development**
|
| 115 |
-
- Humanoid robot control research
|
| 116 |
-
- Whole-body motion learning
|
| 117 |
-
- Sim-to-real transfer studies
|
| 118 |
-
- Teleoperation system development
|
| 119 |
-
|
| 120 |
-
✅ **Robotics Applications**
|
| 121 |
-
- Mobile manipulation tasks
|
| 122 |
-
- Human-robot interaction
|
| 123 |
-
- Demonstration collection
|
| 124 |
-
- Behavior cloning baselines
|
| 125 |
-
|
| 126 |
-
### Out-of-Scope Use
|
| 127 |
-
|
| 128 |
-
❌ This model is **NOT** intended for:
|
| 129 |
-
- Safety-critical applications without additional validation
|
| 130 |
-
- Autonomous operation in uncontrolled environments
|
| 131 |
-
- Direct deployment without proper safety measures
|
| 132 |
-
- Applications requiring certified safety guarantees
|
| 133 |
-
|
| 134 |
-
## Limitations
|
| 135 |
-
|
| 136 |
-
- **Hardware Specific**: Models are tuned for specific humanoid platforms (primarily Unitree G1)
|
| 137 |
-
- **Sim-to-Real Gap**: Performance may vary across different hardware configurations
|
| 138 |
-
- **Safety**: Requires proper safety protocols during deployment
|
| 139 |
-
- **Generalization**: Best performance on behaviors similar to training distribution
|
| 140 |
-
- **Environment**: Designed for structured indoor environments
|
| 141 |
-
|
| 142 |
-
## Hardware Requirements
|
| 143 |
-
|
| 144 |
-
### Deployment Platform
|
| 145 |
-
- **Recommended**: NVIDIA Jetson Orin (32GB or 64GB)
|
| 146 |
-
- **Minimum**: NVIDIA GPU with CUDA support
|
| 147 |
-
- **Robot**: Compatible humanoid platforms (Unitree G1, H1, etc.)
|
| 148 |
|
| 149 |
-
|
| 150 |
-
- **GPU**: NVIDIA GPU with 16GB+ VRAM (for training)
|
| 151 |
-
- **RAM**: 32GB+ recommended
|
| 152 |
-
- **Storage**: 100GB+ for datasets and checkpoints
|
| 153 |
|
| 154 |
## Documentation
|
| 155 |
|
|
|
|
| 55 |
- Injured walking
|
| 56 |
- Boxing stance
|
| 57 |
|
| 58 |
+
## VR Whole-Body Teleoperation
|
| 59 |
+
|
| 60 |
+
SONIC supports real-time whole-body teleoperation via PICO VR headset, enabling natural human-to-robot motion transfer for data collection and interactive control.
|
| 61 |
+
|
| 62 |
+
<div align="center">
|
| 63 |
+
<table>
|
| 64 |
+
<tr>
|
| 65 |
+
<td align="center"><b>Walking</b></td>
|
| 66 |
+
<td align="center"><b>Running</b></td>
|
| 67 |
+
</tr>
|
| 68 |
+
<tr>
|
| 69 |
+
<td align="center"><img src="media/teleop_walking.gif" width="400"></td>
|
| 70 |
+
<td align="center"><img src="media/teleop_running.gif" width="400"></td>
|
| 71 |
+
</tr>
|
| 72 |
+
<tr>
|
| 73 |
+
<td align="center"><b>Sideways Movement</b></td>
|
| 74 |
+
<td align="center"><b>Kneeling</b></td>
|
| 75 |
+
</tr>
|
| 76 |
+
<tr>
|
| 77 |
+
<td align="center"><img src="media/teleop_sideways.gif" width="400"></td>
|
| 78 |
+
<td align="center"><img src="media/teleop_kneeling.gif" width="400"></td>
|
| 79 |
+
</tr>
|
| 80 |
+
<tr>
|
| 81 |
+
<td align="center"><b>Getting Up</b></td>
|
| 82 |
+
<td align="center"><b>Jumping</b></td>
|
| 83 |
+
</tr>
|
| 84 |
+
<tr>
|
| 85 |
+
<td align="center"><img src="media/teleop_getup.gif" width="400"></td>
|
| 86 |
+
<td align="center"><img src="media/teleop_jumping.gif" width="400"></td>
|
| 87 |
+
</tr>
|
| 88 |
+
<tr>
|
| 89 |
+
<td align="center"><b>Bimanual Manipulation</b></td>
|
| 90 |
+
<td align="center"><b>Object Hand-off</b></td>
|
| 91 |
+
</tr>
|
| 92 |
+
<tr>
|
| 93 |
+
<td align="center"><img src="media/teleop_bimanual.gif" width="400"></td>
|
| 94 |
+
<td align="center"><img src="media/teleop_switch_hands.gif" width="400"></td>
|
| 95 |
+
</tr>
|
| 96 |
+
</table>
|
| 97 |
+
</div>
|
| 98 |
+
|
| 99 |
+
## Kinematic Planner
|
| 100 |
+
|
| 101 |
+
SONIC includes a kinematic planner for real-time locomotion generation — choose a movement style, steer with keyboard/gamepad, and adjust speed and height on the fly.
|
| 102 |
+
|
| 103 |
+
<div align="center">
|
| 104 |
+
<table>
|
| 105 |
+
<tr>
|
| 106 |
+
<td align="center" colspan="2"><b>In-the-Wild Navigation</b></td>
|
| 107 |
+
</tr>
|
| 108 |
+
<tr>
|
| 109 |
+
<td align="center" colspan="2"><img src="media/planner/planner_in_the_wild_navigation.gif" width="800"></td>
|
| 110 |
+
</tr>
|
| 111 |
+
<tr>
|
| 112 |
+
<td align="center"><b>Run</b></td>
|
| 113 |
+
<td align="center"><b>Happy</b></td>
|
| 114 |
+
</tr>
|
| 115 |
+
<tr>
|
| 116 |
+
<td align="center"><img src="media/planner/planner_run.gif" width="400"></td>
|
| 117 |
+
<td align="center"><img src="media/planner/planner_happy.gif" width="400"></td>
|
| 118 |
+
</tr>
|
| 119 |
+
<tr>
|
| 120 |
+
<td align="center"><b>Stealth</b></td>
|
| 121 |
+
<td align="center"><b>Injured</b></td>
|
| 122 |
+
</tr>
|
| 123 |
+
<tr>
|
| 124 |
+
<td align="center"><img src="media/planner/planner_stealth.gif" width="400"></td>
|
| 125 |
+
<td align="center"><img src="media/planner/planner_injured.gif" width="400"></td>
|
| 126 |
+
</tr>
|
| 127 |
+
<tr>
|
| 128 |
+
<td align="center"><b>Kneeling</b></td>
|
| 129 |
+
<td align="center"><b>Hand Crawling</b></td>
|
| 130 |
+
</tr>
|
| 131 |
+
<tr>
|
| 132 |
+
<td align="center"><img src="media/planner/planner_kneeling.gif" width="400"></td>
|
| 133 |
+
<td align="center"><img src="media/planner/planner_hand_crawling.gif" width="400"></td>
|
| 134 |
+
</tr>
|
| 135 |
+
<tr>
|
| 136 |
+
<td align="center"><b>Elbow Crawling</b></td>
|
| 137 |
+
<td align="center"><b>Boxing</b></td>
|
| 138 |
+
</tr>
|
| 139 |
+
<tr>
|
| 140 |
+
<td align="center"><img src="media/planner/planner_elbow_crawling.gif" width="400"></td>
|
| 141 |
+
<td align="center"><img src="media/planner/planner_boxing.gif" width="400"></td>
|
| 142 |
+
</tr>
|
| 143 |
+
</table>
|
| 144 |
+
</div>
|
| 145 |
+
|
| 146 |
## Quick Start
|
| 147 |
|
| 148 |
📚 **See the [Quick Start Guide](https://zhengyiluo.github.io/GR00T-WholeBodyControl/getting_started/quickstart.html)** for step-by-step instructions on:
|
|
|
|
| 159 |
|
| 160 |
## Model Checkpoints
|
| 161 |
|
| 162 |
+
Pre-trained SONIC policy checkpoints (ONNX format) are available in the `policy/release/` directory of this repository. The models are optimized for real-time inference on NVIDIA Jetson platforms with low-latency control and robust sim-to-real transfer.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 163 |
|
| 164 |
+
See the [Installation Guide](https://zhengyiluo.github.io/GR00T-WholeBodyControl/getting_started/installation_deploy.html) for instructions on downloading and using the model checkpoints.
|
|
|
|
|
|
|
|
|
|
| 165 |
|
| 166 |
## Documentation
|
| 167 |
|