| --- |
| language: |
| - en |
| library_name: lerobot |
| pipeline_tag: robotics |
| tags: |
| - vision-language-action |
| - imitation-learning |
| - lerobot |
| - pi05 |
| - manipulation |
| inference: false |
| datasets: |
| - PickNikRobotics/kinova_gen3_cube_stack_sim |
| base_model: |
| - lerobot/pi05_base |
| license: gemma |
| extra_gated_heading: Access this model on Hugging Face |
| extra_gated_prompt: >- |
| This model is a Gemma Model Derivative and is provided under and subject to the |
| Gemma Terms of Use (ai.google.dev/gemma/terms). Its use is restricted by the |
| Gemma Prohibited Use Policy (ai.google.dev/gemma/prohibited_use_policy). To |
| access it, review and agree to those terms. |
| extra_gated_button_content: Acknowledge license |
| --- |
| |
| # pi0.5: Kinova Gen3 cube stacking |
|
|
| Color-conditioned cube stacking on a Kinova Gen3. Given a prompt such as |
| `stack the red cube on the green cube`, the policy picks the named cube and places it on the |
| named target. |
|
|
| ## License |
|
|
| [Gemma Terms of Use](https://ai.google.dev/gemma/terms). `model.safetensors` contains modified |
| Gemma weights, derived from pi0.5's PaliGemma backbone and Gemma action expert, and is |
| distributed with [`LICENSE_GEMMA.txt`](LICENSE_GEMMA.txt) and [`NOTICE`](NOTICE). |
|
|
| Use is restricted by the |
| [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy). Pass these |
| terms on to anyone you redistribute the weights to. |
|
|
| ## Model description |
|
|
| - **Inputs:** three 480x480 RGB views keyed `scene`, `wrist`, and `overview`, an 8-dimensional |
| joint state, and a language instruction |
| - **Outputs:** 8-dimensional joint position targets: 7 Kinova Gen3 arm joints and the Robotiq |
| 2F-85 gripper |
| - **Training objective:** flow matching |
| - **Control rate:** 10 Hz, action chunks of 50 |
|
|
| Serving needs `--state-dim 8`. `config.json` declares `observation.state` as `[32]`, pi0.5's |
| padded architecture width, while the normalizer statistics are 8-wide. |
|
|
| ## Training |
|
|
| `lerobot-train`, pi0.5 LoRA-backbone: LoRA (r=32, alpha=64) on the PaliGemma language model's |
| attention and MLP projections, dense training for the action expert and projection heads, SigLIP |
| frozen. 50,000 steps, batch 4, lr 2.5e-5 cosine, weight decay 0.01, seed 1000, bfloat16. See |
| `train_config.json` for the full configuration. |
|
|
| Trained on |
| [`kinova_gen3_cube_stack_sim`](https://huggingface.co/datasets/PickNikRobotics/kinova_gen3_cube_stack_sim), |
| 360 episodes across all six color-ordered prompts. |
|
|
| ## Evaluation |
|
|
| 150 held-out layouts in MuJoCo, real-time chunking on. |
|
|
| | gate | rate | |
| |---|---| |
| | **pick**, the named cube grasped and lifted | 94.0% | |
| | **stack**, placed on the named target | 81.3% | |
|
|
