README.md

---
license: other
license_name: nvidia-open-model-license
license_link: LICENSE
tags:
- robotics
- humanoid
- whole-body-control
- reinforcement-learning
- motion-tracking
- teleoperation
- pytorch
- isaac-lab
pipeline_tag: reinforcement-learning
---

# GEAR-SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control

<div align="center">
  <img src="sonic-preview-gif-480P.gif" width="800">
</div>

## Model Description

**SONIC** (Supersizing Motion Tracking) is a humanoid behavior foundation model developed by NVIDIA that gives robots a core set of motor skills learned from large-scale human motion data. Rather than building separate controllers for predefined motions, SONIC uses motion tracking as a scalable training task, enabling a single unified policy to produce natural, whole-body movement and support a wide range of behaviors.

### Key Features

- 🤖 **Unified Whole-Body Control**: Single policy handles walking, running, crawling, jumping, manipulation, and more
- 🎯 **Motion Tracking**: Trained on large-scale human motion data for natural movements
- 🎮 **Real-Time Teleoperation**: VR-based whole-body teleoperation via PICO headset
- 🚀 **Hardware Deployment**: C++ inference stack for real-time control on humanoid robots
- 🎨 **Kinematic Planner**: Real-time locomotion generation with multiple movement styles
- 🔄 **Multi-Modal Control**: Supports keyboard, gamepad, VR, and high-level planning

## VR Whole-Body Teleoperation

SONIC supports real-time whole-body teleoperation via PICO VR headset, enabling natural human-to-robot motion transfer for data collection and interactive control.

<div align="center">
<table>
<tr>
<td align="center"><b>Walking</b></td>
<td align="center"><b>Running</b></td>
</tr>
<tr>
<td align="center"><img src="media/teleop_walking.gif" width="400"></td>
<td align="center"><img src="media/teleop_running.gif" width="400"></td>
</tr>
<tr>
<td align="center"><b>Sideways Movement</b></td>
<td align="center"><b>Kneeling</b></td>
</tr>
<tr>
<td align="center"><img src="media/teleop_sideways.gif" width="400"></td>
<td align="center"><img src="media/teleop_kneeling.gif" width="400"></td>
</tr>
<tr>
<td align="center"><b>Getting Up</b></td>
<td align="center"><b>Jumping</b></td>
</tr>
<tr>
<td align="center"><img src="media/teleop_getup.gif" width="400"></td>
<td align="center"><img src="media/teleop_jumping.gif" width="400"></td>
</tr>
<tr>
<td align="center"><b>Bimanual Manipulation</b></td>
<td align="center"><b>Object Hand-off</b></td>
</tr>
<tr>
<td align="center"><img src="media/teleop_bimanual.gif" width="400"></td>
<td align="center"><img src="media/teleop_switch_hands.gif" width="400"></td>
</tr>
</table>
</div>

## Kinematic Planner

SONIC includes a kinematic planner for real-time locomotion generation — choose a movement style, steer with keyboard/gamepad, and adjust speed and height on the fly.

<div align="center">
<table>
<tr>
<td align="center" colspan="2"><b>In-the-Wild Navigation</b></td>
</tr>
<tr>
<td align="center" colspan="2"><img src="media/planner/planner_in_the_wild_navigation.gif" width="800"></td>
</tr>
<tr>
<td align="center"><b>Run</b></td>
<td align="center"><b>Happy</b></td>
</tr>
<tr>
<td align="center"><img src="media/planner/planner_run.gif" width="400"></td>
<td align="center"><img src="media/planner/planner_happy.gif" width="400"></td>
</tr>
<tr>
<td align="center"><b>Stealth</b></td>
<td align="center"><b>Injured</b></td>
</tr>
<tr>
<td align="center"><img src="media/planner/planner_stealth.gif" width="400"></td>
<td align="center"><img src="media/planner/planner_injured.gif" width="400"></td>
</tr>
<tr>
<td align="center"><b>Kneeling</b></td>
<td align="center"><b>Hand Crawling</b></td>
</tr>
<tr>
<td align="center"><img src="media/planner/planner_kneeling.gif" width="400"></td>
<td align="center"><img src="media/planner/planner_hand_crawling.gif" width="400"></td>
</tr>
<tr>
<td align="center"><b>Elbow Crawling</b></td>
<td align="center"><b>Boxing</b></td>
</tr>
<tr>
<td align="center"><img src="media/planner/planner_elbow_crawling.gif" width="400"></td>
<td align="center"><img src="media/planner/planner_boxing.gif" width="400"></td>
</tr>
</table>
</div>

## Quick Start

📚 **See the [Quick Start Guide](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/quickstart.html)** for step-by-step instructions on:
- Installation and setup
- Running SONIC with different control modes (keyboard, gamepad, VR)
- Deploying on real hardware
- Using the kinematic planner

**Key Resources:**
- [Installation Guide](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/installation_deploy.html) - Complete setup instructions
- [Keyboard Control Tutorial](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/keyboard.html) - Get started with keyboard control
- [Gamepad Control Tutorial](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/gamepad.html) - Set up gamepad control
- [VR Teleoperation Setup](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/vr_teleop_setup.html) - Full-body VR control

## Model Checkpoints

The repository provides three Unitree G1 controller variants. Each ONNX
encoder must be used with the decoder and observation config from the same
variant.

| Model | Location | SMPL reference input | Intended use and comments |
|---|---|---|---|
| **Default SONIC (original release)** | Top-level deployment files; PyTorch checkpoint under `sonic_release/` | 10 frames at 20 ms spacing, approximately 200 ms | General-purpose motion tracking, planning, and teleoperation. G1 and teleoperation references use `step5`. |
| **Low-latency teleoperation** | `low_latency/` | 4 frames at 20 ms spacing, approximately 80 ms | Reduced-reference-lookahead controller for responsive teleoperation and VLA execution. G1 and teleoperation references use `step1`. |
| **SONIC v1.1** | `sonic_v1_1/` | 10 frames at 20 ms spacing, approximately 200 ms | Robot-heading-normalized controller trained with wrist-pose augmentation for heading-stable 3-point teleoperation and SONIC-backed VLA policies. G1 and teleoperation references use `step5`; this is not the low-latency model. |

All variants use 64-dimensional universal motion tokens and a 29-DoF action
decoder. Inference is powered by TensorRT and runs on both desktop and Jetson
hardware.

| Checkpoint | File | Description |
|---|---|---|
| Policy encoder | `model_encoder.onnx` | Encodes motion reference into latent |
| Policy decoder | `model_decoder.onnx` | Decodes latent into joint actions |
| Kinematic planner | `planner_sonic.onnx` | Real-time locomotion style planner |
| Low-latency policy | `low_latency/model_encoder.onnx`, `low_latency/model_decoder.onnx`, `low_latency/observation_config.yaml` | Reduced-lookahead G1 controller variant |
| Low-latency PyTorch checkpoint | `low_latency/last.pt` | Training checkpoint and config for the low-latency variant |
| SONIC v1.1 policy | `sonic_v1_1/model_encoder.onnx`, `sonic_v1_1/model_decoder.onnx`, `sonic_v1_1/observation_config.yaml` | Robot-heading-normalized G1 controller variant |
| SONIC v1.1 PyTorch checkpoint | `sonic_v1_1/last.pt` | Training checkpoint and configs for SONIC v1.1 |

**Quick download** (requires `pip install huggingface_hub`):

```python
from huggingface_hub import snapshot_download
snapshot_download(repo_id="nvidia/GEAR-SONIC", local_dir="gear_sonic_deploy")
```

Or use the download script from the GitHub repo:

```bash
python download_from_hf.py             # policy + planner (default)
python download_from_hf.py --no-planner # policy only
python download_from_hf.py --low-latency # low-latency policy + planner
python download_from_hf.py --sonic-v1-1 # SONIC v1.1 policy + planner
```

See the [Download Models guide](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/download_models.html) for full instructions.

## SONIC v1.1 Inference

The `sonic_v1_1/` variant uses robot-heading-normalized target orientation
and a 10-frame SMPL/wrist reference horizon. It was trained with wrist-pose
augmentation and is intended for 3-point teleoperation and SONIC-backed VLA
policies trained against this controller.

```bash
git clone https://github.com/NVlabs/GR00T-WholeBodyControl.git
cd GR00T-WholeBodyControl
pip install huggingface_hub
python download_from_hf.py --sonic-v1-1

cd gear_sonic_deploy
./deploy.sh \
    --cp policy/sonic_v1_1/model \
    --obs-config policy/sonic_v1_1/observation_config.yaml \
    sim
```

For the Python VLA launcher, pass the same two paths with
`--deploy-checkpoint` and `--deploy-obs-config`.

## Low-Latency Inference

The low-latency SONIC variant is stored under `low_latency/` and does not
replace the default top-level policy. To run it with the C++ deployment stack:

```bash
git clone https://github.com/NVlabs/GR00T-WholeBodyControl.git
cd GR00T-WholeBodyControl
pip install huggingface_hub
python download_from_hf.py --low-latency
```

### C++ deployment

Simulation:

```bash
cd gear_sonic_deploy
./deploy.sh \
    --cp policy/low_latency/model \
    --obs-config policy/low_latency/observation_config.yaml \
    sim
```

Real robot VLA or teleoperation:

```bash
cd gear_sonic_deploy
./deploy.sh \
    --cp policy/low_latency/model \
    --obs-config policy/low_latency/observation_config.yaml \
    --input-type zmq_manager \
    real
```

`deploy.sh` expects `--cp` to be the shared model prefix and appends
`_encoder.onnx` and `_decoder.onnx` internally. For the full VLA deployment
flow, see the [VLA Inference guide](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/vla_inference.html).

### Python launcher

The Python VLA launcher can start the same low-latency C++ deployment pane and
run the Python VLA client, keyboard publisher, and optional data exporter:

```bash
python gear_sonic/scripts/launch_inference.py \
    --deploy-checkpoint policy/low_latency/model \
    --deploy-obs-config policy/low_latency/observation_config.yaml \
    --camera-host 192.168.123.164 \
    --prompt "pick up the cup"
```

### Python checkpoint evaluation

For Isaac Lab evaluation or rendering from the released PyTorch checkpoint:

```bash
python download_from_hf.py --training --low-latency
python download_from_hf.py --sample

python gear_sonic/eval_agent_trl.py \
    +checkpoint=low_latency/last.pt \
    +headless=False \
    ++num_envs=1 \
    ++manager_env.observations.policy.enable_corruption=False \
    ++manager_env.observations.tokenizer.enable_corruption=False \
    "++manager_env.commands.motion.motion_lib_cfg.motion_file=sample_data/robot_filtered" \
    "++manager_env.commands.motion.motion_lib_cfg.smpl_motion_file=sample_data/smpl_filtered"
```

## Documentation

📚 **[Full Documentation](https://nvlabs.github.io/GR00T-WholeBodyControl/)**

### Guides
- [Installation (Deployment)](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/installation_deploy.html)
- [Installation (Training)](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/installation_training.html)
- [Quick Start](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/quickstart.html)
- [VR Teleoperation Setup](https://nvlabs.github.io/GR00T-WholeBodyControl/getting_started/vr_teleop_setup.html)

### Tutorials
- [Keyboard Control](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/keyboard.html)
- [Gamepad Control](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/gamepad.html)
- [VR Whole-Body Teleoperation](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/vr_wholebody_teleop.html)

## Repository Structure

```
GR00T-WholeBodyControl/
├── gear_sonic_deploy/     # C++ inference stack for deployment
├── gear_sonic/            # Teleoperation and data collection tools
├── decoupled_wbc/         # Decoupled WBC (GR00T N1.5/N1.6)
├── docs/                  # Documentation source
└── media/                 # Videos and images
```

## Related Projects

This repository is part of NVIDIA's GR00T (Generalist Robot 00 Technology) initiative:
- **[GR00T N1.5](https://research.nvidia.com/labs/gear/gr00t-n1_5/)**: Previous generation decoupled controller
- **[GR00T N1.6](https://research.nvidia.com/labs/gear/gr00t-n1_6/)**: Improved decoupled WBC approach
- **[GEAR-SONIC Website](https://nvlabs.github.io/GEAR-SONIC/)**: Project page with videos and details

## Citation

If you use GEAR-SONIC in your research, please cite:

```bibtex
@article{luo2025sonic,
    title={SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control},
    author={Luo, Zhengyi and Yuan, Ye and Wang, Tingwu and Li, Chenran and Chen, Sirui and Casta\~neda, Fernando and Cao, Zi-Ang and Li, Jiefeng and Minor, David and Ben, Qingwei and Da, Xingye and Ding, Runyu and Hogg, Cyrus and Song, Lina and Lim, Edy and Jeong, Eugene and He, Tairan and Xue, Haoru and Xiao, Wenli and Wang, Zi and Yuen, Simon and Kautz, Jan and Chang, Yan and Iqbal, Umar and Fan, Linxi and Zhu, Yuke},
    journal={arXiv preprint arXiv:2511.07820},
    year={2025}
}
```

## License

This project uses **dual licensing**:

- **Source Code**: Apache License 2.0 - applies to all code, scripts, and software components
- **Model Weights**: NVIDIA Open Model License - applies to all trained model checkpoints

**Key points of the NVIDIA Open Model License:**
- ✅ Commercial use permitted with attribution
- ✅ Modification and distribution allowed
- ⚠️ Must comply with NVIDIA's Trustworthy AI terms
- ⚠️ Model outputs subject to responsible use guidelines

See [LICENSE](https://github.com/NVlabs/GR00T-WholeBodyControl/blob/main/LICENSE) for complete terms.

## Support & Contact

- 📧 **Email**: [gear-wbc@nvidia.com](mailto:gear-wbc@nvidia.com)
- 🐛 **Issues**: [GitHub Issues](https://github.com/NVlabs/GR00T-WholeBodyControl/issues)
- 📖 **Documentation**: [https://nvlabs.github.io/GR00T-WholeBodyControl/](https://nvlabs.github.io/GR00T-WholeBodyControl/)
- 🌐 **Website**: [https://nvlabs.github.io/GEAR-SONIC/](https://nvlabs.github.io/GEAR-SONIC/)

## Acknowledgments

This work builds upon and acknowledges:
- [Beyond Mimic](https://github.com/HybridRobotics/whole_body_tracking) - Whole-body tracking foundation
- [Isaac Lab](https://github.com/isaac-sim/IsaacLab) - Robot learning framework
- NVIDIA Research GEAR Lab team
- All contributors and collaborators

## Model Card Contact

For questions about this model card or responsible AI considerations, contact: [gear-wbc@nvidia.com](mailto:gear-wbc@nvidia.com)