Update README.md

README.md

@@ -1,69 +1,66 @@
-# π₀.₅ - Base
+# π₀.₅ (Pi05) Policy
 
-This is a PyTorch version of the PI0.5 `pi05_base model`, converted from the original JAX/Flax implementation.
+π₀.₅ is a **Vision-Language-Action model with open-world generalization**, from Physical Intelligence. The LeRobot implementation is adapted from their open source [OpenPI](https://github.com/Physical-Intelligence/openpi) repository.
 
-## Model Details
+## Model Overview
 
-- **Architecture**: PI0.5 (Vision-Language-Action model with discrete state input)
-- **Model Type**: PI0.5
-- **Domain**: Base model (general purpose)
-- **Precision**: 32-bit floating point (fp32)
-- **Vision Model**: PaliGemma (gemma_2b)
-- **Action Expert**: gemma_300m
+π₀.₅ represents a significant evolution from π₀, developed by [Physical Intelligence](https://www.physicalintelligence.company/blog/pi05) to address a big challenge in robotics: **open-world generalization**. While robots can perform impressive tasks in controlled environments, π₀.₅ is designed to generalize to entirely new environments and situations that were never seen during training.
 
-## Key Features
+### The Generalization Challenge
 
-- **Discrete State Input**: Uses discrete language tokens for state representation
-- **Flow Matching**: Utilizes adaRMSNorm for timestep injection in action expert
-- **Enhanced Action Modeling**: Improved action prediction with flow matching approach
+As Physical Intelligence explains, the fundamental challenge isn't performing tasks of agility or dexterity, but generalization, the ability to correctly perform tasks in new settings with new objects. Consider a robot cleaning different homes: each home has different objects in different places. Generalization must occur at multiple levels:
 
-## Conversion Details
+- **Physical Level**: Understanding how to pick up a spoon (by the handle) or plate (by the edge), even with unseen objects in cluttered environments
+- **Semantic Level**: Understanding task semantics, where to put clothes and shoes (laundry hamper, not on the bed), and what tools are appropriate for cleaning spills
+- **Environmental Level**: Adapting to "messy" real-world environments like homes, grocery stores, offices, and hospitals
 
-This model was converted from JAX to PyTorch using the OpenPI conversion script:
+### Co-Training on Heterogeneous Data
 
-```bash
-python examples/convert_jax_model_to_pytorch.py \
-    --checkpoint_dir /pi05_base \
-    --config_name pi05_base \
-    --output_path /pi05_base/pytorch/fp32/ \
-    --precision float32
-```
+The breakthrough innovation in π₀.₅ is **co-training on heterogeneous data sources**. The model learns from:
 
-## Usage
+1. **Multimodal Web Data**: Image captioning, visual question answering, object detection
+2. **Verbal Instructions**: Humans coaching robots through complex tasks step-by-step
+3. **Subtask Commands**: High-level semantic behavior labels (e.g., "pick up the pillow" for an unmade bed)
+4. **Cross-Embodiment Robot Data**: Data from various robot platforms with different capabilities
+5. **Multi-Environment Data**: Static robots deployed across many different homes
+6. **Mobile Manipulation Data**: ~400 hours of mobile robot demonstrations
 
-```python
-from openpi.models_pytorch.pi0_pytorch import PI0Pytorch
-import torch
+This diverse training mixture creates a "curriculum" that enables generalization across physical, visual, and semantic levels simultaneously.
 
-# Load the model
-model = PI0Pytorch.from_pretrained("pepijn223/pi05_base_fp32")
-
-# The model expects inputs in the format:
-# - images: torch.Tensor of shape [batch, height, width, channels]  
-# - text: tokenized text prompts
-# - proprioceptive_state: robot state information (if applicable)
-```
 
-## Model Architecture
+## Training
 
-The model consists of:
-1. **Vision Encoder**: PaliGemma-based vision processing
-2. **Language Encoder**: Text prompt understanding  
-3. **Action Expert**: Specialized network for action prediction
-4. **Integration Layer**: Combines multimodal information for action output
+Here's a complete training command for finetuning the base π₀.₅ model on your own dataset:
 
-## Training Data
+```bash
+python src/lerobot/scripts/train.py \
+    --dataset.repo_id=your_dataset \
+    --policy.type=pi05 \
+    --output_dir=./outputs/pi05_training \
+    --job_name=pi0_training \
+    --policy.repo_id=pepijn223/pi05_base \
+    --policy.pretrained_path=your_repo_id \
+    --policy.compile_model=true \
+    --policy.gradient_checkpointing=true \
+    --wandb.enable=true \
+    --policy.dtype=bfloat16 \
+    --steps=3000 \
+    --policy.scheduler_decay_steps=3000 \
+    --policy.device=cuda \
+    --batch_size=32
+```
 
-This model was trained on robotics datasets appropriate for its domain:
-- **DROID models**: Trained on diverse robot manipulation data
-- **LIBERO models**: Trained on diverse tabletop manipulation scenarios  
-- **Base models**: Trained on general robotics datasets
+## Conversion Details
 
-## Limitations
+This model was converted from JAX to PyTorch using the OpenPI conversion script:
 
-- Model performance depends on similarity between deployment and training environments
-- May require domain-specific fine-tuning for optimal performance
-- Action space must match the trained action dimension (32)
+```bash
+python examples/convert_jax_model_to_pytorch.py \
+    --checkpoint_dir /pi05_base \
+    --config_name pi05_base \
+    --output_path /pi05_base/pytorch/fp32/ \
+    --precision float32
+```
 
 ## Citation