File size: 4,550 Bytes
8d86e06 39207a8 8d86e06 39207a8 8d86e06 39207a8 8d86e06 39207a8 8d86e06 39207a8 8d86e06 39207a8 8d86e06 39207a8 8d86e06 39207a8 8d86e06 39207a8 8d86e06 39207a8 8d86e06 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
tags:
- Embodied-AI
- Robotics
- Offline-to-Online-RL
- Value-Function
- VLA-Model
- Robot-Foundation-Model
- Vision-Language-Action
- Robot-Learning
- Imitation-Learning
---
# Robo-ValueRL
[[Project Page](https://gewu-lab.github.io/robo-valuerl/)] [[GitHub](https://github.com/Open-X-Humanoid/Robo-ValueRL)] [[Dataset](https://huggingface.co/datasets/X-Humanoid/Robo-ValueRL)] [[Paper](#)]
This repository contains the **model checkpoints** for **Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning**.
Robo-ValueRL studies how reliable value estimation can guide robotic policy learning from heterogeneous offline demonstrations and online rollout trajectories. The framework learns a history-conditioned value estimator, converts value differences into action-quality conditions for offline policy pretraining, and uses value-guided rollout filtering for stable online residual adaptation.
## Model Description
**Robo-ValueRL** is an offline-to-online robotic reinforcement learning framework centered on reliable value estimation. Instead of only reporting final task success, Robo-ValueRL explicitly diagnoses whether learned values capture global task progress and local action-level preference, then propagates these value signals into downstream policy learning.
The released model suite includes:
1. **History-Conditioned Value Estimator**
Predicts normalized task progress from multi-view robot observations, language instructions, and visual history. The temporal context helps reduce ambiguity caused by occlusions, repeated motions, and visually similar task stages.
2. **Quality-Conditioned VLA Policy**
Uses value differences to derive action-quality conditions. These conditions guide a Vision-Language-Action policy during offline pretraining, allowing the policy to prioritize useful behaviors from mixed-quality demonstrations.
3. **Online Residual Adaptation Module**
Learns lightweight corrections from value-filtered online rollouts while keeping the pretrained base policy frozen. This enables targeted failure recovery and self-correction without overwriting the offline prior.
## Model Hierarchy
1. **Value Estimation**
- Learns history-conditioned value functions from heterogeneous robot data.
- Evaluates value reliability with global-progress and local-preference metrics.
2. **Offline Policy Pretraining**
- Converts value differences into action-quality conditions.
- Trains a quality-conditioned VLA policy on mixed-quality demonstrations.
3. **Online Policy Improvement**
- Uses reliable value estimates to filter online rollout data.
- Trains a lightweight residual adapter for targeted real-world improvement.
## Associated Dataset
The models are trained and evaluated with the Robo-ValueRL dataset:
[[Robo-ValueRL Dataset](https://huggingface.co/datasets/X-Humanoid/Robo-ValueRL)]
The dataset contains heterogeneous real-robot demonstrations and online rollout trajectories for chip insertion and block disassembly.
## Key Features
* **Reliable Value Estimation**: Uses visual history to produce smoother progress estimates and sharper error responses.
* **Value-Guided Data Utilization**: Prioritizes useful demonstrations and rollout segments from heterogeneous robot experience.
* **Quality-Conditioned Policy Learning**: Conditions the VLA policy on value-derived action quality.
* **Stable Offline-to-Online Improvement**: Improves real-world performance through residual adaptation while preserving the pretrained base policy.
* **Real-Robot Evaluation**: Evaluated on precision chip insertion and generalizable block disassembly.
## Highlights
- 86% final success on chip insertion
- 84% final success on block disassembly
- +26% offline gain on chip insertion
- +34% offline gain on block disassembly
- 240h offline demonstrations
- 3,000+ online rollout trajectories
## Usage
The released assets are organized for reproducing Robo-ValueRL's model pipeline.
Please refer to the [GitHub repository](https://github.com/Open-X-Humanoid/Robo-ValueRL) for setup instructions, inference scripts, training code, and data-processing utilities.
## Citation
If you use Robo-ValueRL in your research, please cite our work. Citation will be updated after the arXiv release.
## License
Please refer to the license file in the [GitHub repository](https://github.com/Open-X-Humanoid/Robo-ValueRL).
## Contact
For questions, please open an issue on our [GitHub repository](https://github.com/Open-X-Humanoid/Robo-ValueRL). |