Robo-ValueRL

[Project Page] [GitHub] [Paper] [Data]

This repository contains the models and data for Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning.

Robo-ValueRL studies how reliable value estimation can guide robotic policy learning from heterogeneous offline demonstrations and online rollout trajectories. The framework learns a history-conditioned value estimator, converts value differences into action-quality conditions for offline policy pretraining, and uses value-guided rollout filtering for stable online residual adaptation.

🤖 Model Description

Robo-ValueRL is an offline-to-online robotic reinforcement learning framework centered on reliable value estimation. Instead of only reporting final task success, Robo-ValueRL explicitly diagnoses whether learned values capture global task progress and local action-level preference, then propagates these value signals into downstream policy learning.

The released model suite includes:

History-Conditioned Value Estimator
Predicts normalized task progress from multi-view robot observations, language instructions, and visual history. The temporal context helps reduce ambiguity caused by occlusions, repeated motions, and visually similar task stages.
Quality-Conditioned VLA Policy
Uses value differences to derive action-quality conditions. These conditions guide a Vision-Language-Action policy during offline pretraining, allowing the policy to prioritize useful behaviors from mixed-quality demonstrations.
Online Residual Adaptation Module
Learns lightweight corrections from value-filtered online rollouts while keeping the pretrained base policy frozen. This enables targeted failure recovery and self-correction without overwriting the offline prior.

📦 Data Description

The Robo-ValueRL dataset contains heterogeneous real-robot experience collected on two long-horizon manipulation tasks:

Chip Insertion: millimeter-level precision manipulation requiring PCB grasping, pose adjustment, chip grasping, and insertion.
Block Disassembly: generalizable object disassembly requiring robust grasping, separation, and classification behaviors.

The dataset includes:

240 hours of offline demonstrations
3,000+ online rollout trajectories
Multi-view robot observations
Language task instructions
Robot states and action chunks
Mixed-quality trajectories, including successful demonstrations, corrections, suboptimal behaviors, and failure cases
Value-derived action-quality labels / conditions for policy learning
Online rollout segments for residual adaptation

🧱 Model Hierarchy

Value Estimation
- Learns history-conditioned value functions from heterogeneous robot data.
- Evaluates value reliability with global-progress and local-preference metrics.
Offline Policy Pretraining
- Converts value differences into action-quality conditions.
- Trains a quality-conditioned VLA policy on mixed-quality demonstrations.
Online Policy Improvement
- Uses reliable value estimates to filter online rollout data.
- Trains a lightweight residual adapter for targeted real-world improvement.

✨ Key Features

Reliable Value Estimation: Uses visual history to produce smoother progress estimates and sharper error responses.
Value-Guided Data Utilization: Prioritizes useful demonstrations and rollout segments from heterogeneous robot experience.
Quality-Conditioned Policy Learning: Conditions the VLA policy on value-derived action quality.
Stable Offline-to-Online Improvement: Improves real-world performance through residual adaptation while preserving the pretrained base policy.
Real-Robot Evaluation: Evaluated on precision chip insertion and generalizable block disassembly.

📊 Highlights

86% final success on chip insertion
84% final success on block disassembly
+26% offline gain on chip insertion
+34% offline gain on block disassembly
240h offline demonstrations
3,000+ online rollout trajectories

🛠 Usage

The released assets are organized for reproducing Robo-ValueRL's model and data pipeline.

Please refer to the GitHub repository for setup instructions, inference scripts, training code, and data-processing utilities.

📝 Citation

If you use the XR-1 model in your research, please cite our work:(Citation will be updated after the arXiv release.)

📜 License

Please refer to the license file in the GitHub repository.

📬 Contact

For questions, please open an issue on our GitHub repository.