| # Robo-ValueRL |
|
|
| [[Project Page](https://gewu-lab.github.io/robo-valuerl/)] [[GitHub](https://github.com/Open-X-Humanoid/Robo-ValueRL)] [[Paper](#)] [[Data](#)] |
|
|
| This repository contains the **models and data** for **Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning**. |
|
|
| Robo-ValueRL studies how reliable value estimation can guide robotic policy learning from heterogeneous offline demonstrations and online rollout trajectories. The framework learns a history-conditioned value estimator, converts value differences into action-quality conditions for offline policy pretraining, and uses value-guided rollout filtering for stable online residual adaptation. |
|
|
| ## 🤖 Model Description |
|
|
| **Robo-ValueRL** is an offline-to-online robotic reinforcement learning framework centered on reliable value estimation. Instead of only reporting final task success, Robo-ValueRL explicitly diagnoses whether learned values capture global task progress and local action-level preference, then propagates these value signals into downstream policy learning. |
|
|
| The released model suite includes: |
|
|
| 1. **History-Conditioned Value Estimator** |
| Predicts normalized task progress from multi-view robot observations, language instructions, and visual history. The temporal context helps reduce ambiguity caused by occlusions, repeated motions, and visually similar task stages. |
|
|
| 2. **Quality-Conditioned VLA Policy** |
| Uses value differences to derive action-quality conditions. These conditions guide a Vision-Language-Action policy during offline pretraining, allowing the policy to prioritize useful behaviors from mixed-quality demonstrations. |
|
|
| 3. **Online Residual Adaptation Module** |
| Learns lightweight corrections from value-filtered online rollouts while keeping the pretrained base policy frozen. This enables targeted failure recovery and self-correction without overwriting the offline prior. |
|
|
| ## 📦 Data Description |
|
|
| The Robo-ValueRL dataset contains heterogeneous real-robot experience collected on two long-horizon manipulation tasks: |
|
|
| - **Chip Insertion**: millimeter-level precision manipulation requiring PCB grasping, pose adjustment, chip grasping, and insertion. |
| - **Block Disassembly**: generalizable object disassembly requiring robust grasping, separation, and classification behaviors. |
|
|
| The dataset includes: |
|
|
| - **240 hours** of offline demonstrations |
| - **3,000+** online rollout trajectories |
| - Multi-view robot observations |
| - Language task instructions |
| - Robot states and action chunks |
| - Mixed-quality trajectories, including successful demonstrations, corrections, suboptimal behaviors, and failure cases |
| - Value-derived action-quality labels / conditions for policy learning |
| - Online rollout segments for residual adaptation |
|
|
| ## 🧱 Model Hierarchy |
|
|
| 1. **Value Estimation** |
| - Learns history-conditioned value functions from heterogeneous robot data. |
| - Evaluates value reliability with global-progress and local-preference metrics. |
|
|
| 2. **Offline Policy Pretraining** |
| - Converts value differences into action-quality conditions. |
| - Trains a quality-conditioned VLA policy on mixed-quality demonstrations. |
|
|
| 3. **Online Policy Improvement** |
| - Uses reliable value estimates to filter online rollout data. |
| - Trains a lightweight residual adapter for targeted real-world improvement. |
|
|
| ## ✨ Key Features |
|
|
| * **Reliable Value Estimation**: Uses visual history to produce smoother progress estimates and sharper error responses. |
| * **Value-Guided Data Utilization**: Prioritizes useful demonstrations and rollout segments from heterogeneous robot experience. |
| * **Quality-Conditioned Policy Learning**: Conditions the VLA policy on value-derived action quality. |
| * **Stable Offline-to-Online Improvement**: Improves real-world performance through residual adaptation while preserving the pretrained base policy. |
| * **Real-Robot Evaluation**: Evaluated on precision chip insertion and generalizable block disassembly. |
|
|
| ## 📊 Highlights |
|
|
| - 86% final success on chip insertion |
| - 84% final success on block disassembly |
| - +26% offline gain on chip insertion |
| - +34% offline gain on block disassembly |
| - 240h offline demonstrations |
| - 3,000+ online rollout trajectories |
|
|
| ## 🛠 Usage |
|
|
| The released assets are organized for reproducing Robo-ValueRL's model and data pipeline. |
|
|
| Please refer to the [GitHub repository](https://github.com/Open-X-Humanoid/Robo-ValueRL) for setup instructions, inference scripts, training code, and data-processing utilities. |
|
|
| ## 📝 Citation |
| If you use the XR-1 model in your research, please cite our work:(Citation will be updated after the arXiv release.) |
|
|
| ## 📜 License |
| Please refer to the license file in the [GitHub repository](https://github.com/Open-X-Humanoid/Robo-ValueRL). |
|
|
| ## 📬 Contact |
| For questions, please open an issue on our [GitHub repository](https://github.com/Open-X-Humanoid/Robo-ValueRL). |
|
|
