README.md

> [!IMPORTANT]
> 🌟 Stay up to date at [opendrivelab.com](https://opendrivelab.com/#news)!

<h1 align="center"> R2SE: Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving </h1> 

<div align="center">
 
[![arXiv](https://img.shields.io/badge/arXiv-2506.09800-479ee2.svg)](https://arxiv.org/abs/2506.09800)

</div>

<div id="top" align="center">
<p align="center">
<img src="assets/r2se_teaser.png" width="1000px" >
</p>
</div>

> - [Haochen Liu](https://scholar.google.com/citations?user=iizqKUsAAAAJ&hl), [Tianyu Li](https://scholar.google.com/citations?user=X6vTmEMAAAAJ&hl=zh-CN), Haohan Yang, Li Chen, Caojun Wang, Ke Guo, Haochen Tian, Hongchen Li, Hongyang Li and Chen Lv
> - [Paper](https://arxiv.org/abs/2506.09800) 

> - If you have any questions, please feel free to contact: *Haochen Liu* ( haochen002@e.ntu.edu.sg )


---

**[2025-11]** Initial project released.

<!-- Full code and checkpoints release is coming soon. Please stay tuned. -->

## Overview

**R2SE** is a RL finetuning framework targeting hard cases and forgetting in End-to-end Autonmous Driving; 

<div id="top" align="center">
<p align="center">
<img src="assets/r2se.png" width="1000px" >
</p>
</div>


<!-- <div id="top" align="center">
<p align="center">
<img src="assets/betopnet.png" width="1000px" >
</p>
</div> -->

## Get Started

### Data Preparation

We follow the PARA-Drive pipeline in formating train/test data.

```
python scripts/data_converter/nuplan/e2e_nuplan_openscenes_navsim_multiprocessing.py
```


### R2SE-GP Training

Following the pipeline in UniAD using specific configs in ```config/paradrive```

1. Pretrain the perception / backbone:

```
configs/paradrive/exp_100pct/e2e_r2se_gp_perception.py
```

2. Pretrain the full R2SE-GP:

```
configs/paradrive/exp_100pct/e2e_r2se_gp.py
```

### R2SE RL Finetuning

1. Hard case sampling: User may rerun the navsim inference and sort for hard case tokens, we provide some preprocessed ones in ```/data_yaml``` for your convenience.

2. RL Finetuning:

```
configs/paradrive/exp_100pct/e2e_r2se.py
```

After finetuning, run the testing script for both ```R2SE-GP``` and ```R2SE```, you are expected to get two ```.pkl``` for saved token and planning traj.

3. OOD Inference:

Load ```.pkl``` from ```R2SE``. Run this script to get the switched planning results

```
python ood_inference.py
```

### Testing

run the script in ```navsim```, ensure you load all the cached data and inference result of  ```R2SE-GP``` in ```.csv``` format for PDMS and Forgetting metric calculations.

### Weights

All weights are provided in ```unknownuser6666/R2SE_weights```

## TODO List

- [x] Initial release
- [] Code Reorganization


## Citation

If you find the project helpful for your research, please consider citing our paper:

```bibtex
@article{liu2025reinforced,
  title={Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving},
  author={Liu, Haochen and Li, Tianyu and Yang, Haohan and Chen, Li and Wang, Caojun and Guo, Ke and Tian, Haochen and Li, Hongchen and Li, Hongyang and Lv, Chen},
  journal={arXiv preprint arXiv:2506.09800},
  year={2025}
}
```