File size: 3,217 Bytes
a22c98e 3cf807a a22c98e 3cf807a a22c98e 3cf807a a22c98e 3cf807a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
---
base_model:
- Qwen/Qwen2.5-VL-7B-Instruct
datasets:
- WaltonFuture/Multimodal-Cold-Start
- WaltonFuture/Multimodal-RL-Data
license: apache-2.0
pipeline_tag: image-text-to-text
library_name: transformers
---
* 🐙 **GitHub Repo:** [waltonfuture/RL-with-Cold-Start](https://github.com/waltonfuture/RL-with-Cold-Start)
* 📜 **Paper (arXiv):** [Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start (arXiv:2505.22334)](https://arxiv.org/abs/2505.22334)
<div align=center>
<img src="assets/model_comparison.png" width = "80%" alt="model_comparison" align=center/>
</div>
## Cold Start Stage
We conduct supervised fine-tuning on Qwen2.5-VL-3B and Qwen2.5-VL-7B using [ms-swift](https://github.com/modelscope/ms-swift). In this stage, please refer to this curated [dataset](https://huggingface.co/datasets/WaltonFuture/Multimodal-Cold-Start) distilled from Qwen2.5-VL-32B using rejection sampling.
### Setup
```bash
git clone https://github.com/waltonfuture/RL-with-Cold-Start.git
cd RL-with-Cold-Start/SFT
pip install -e .
```
### Prepare Data
```bash
python convert_data.py
```
### SFT
```bash
bash qwen2.5vl_sft.sh
```
The checkpoint can be found in SFT/output.
## RL Stage
We further conduct GRPO using [EasyR1](https://github.com/hiyouga/EasyR1). Please refer to this [dataset](https://huggingface.co/datasets/WaltonFuture/Multimodal-RL-Data) for the GRPO training.
### Setup
```bash
git clone https://github.com/waltonfuture/RL-with-Cold-Start.git
cd RL-with-Cold-Start/GRPO
pip install -e .
```
### GRPO Training (replace the checkpoint with the model after SFT)
```bash
bash examples/qwen2_5_vl_7b_grpo.sh
```
### Merge Checkpoint in Hugging Face Format
```bash
python3 scripts/model_merger.py --local_dir checkpoints/easyr1/qwen2_5_vl_7b_grpo/global_step_80/actor
```
## Data Access
Our two stage datasets are now available on Huggingface.
| Stage | Data |
| ------------------ | ------------- |
| Cold Start | [Multimodal-Cold-Start](https://huggingface.co/datasets/WaltonFuture/Multimodal-Cold-Start) |
| RL | [Multimodal-RL-Data](https://huggingface.co/datasets/WaltonFuture/Multimodal-RL-Data) |
## Model Access
Our models are now available on Huggingface.
| Backbone | Our model |
| ------------------ | ------------- |
| Qwen2.5-VL-7b | [Qwen2.5VL-7b-RL-with-Cold-Start](https://huggingface.co/WaltonFuture/Qwen2.5VL-7b-RLCS) |
| Qwen2.5-VL-3b | [Qwen2.5VL-3b-RL-with-Cold-Start](https://huggingface.co/WaltonFuture/Qwen2.5VL-3b-RLCS) |
## Acknowledgment
Our models are built upon the amazing [Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl-6795ffac22b334a837c0f9a5) family.
We thank [EasyR1](https://github.com/hiyouga/EasyR1) and [ms-swift](https://github.com/modelscope/ms-swift) for their training codes.
## Contact
Please contact Lai Wei (waltonfuture@sjtu.edu.cn) if needed.
## Citation
```
@article{wei2025advancing,
title={Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start},
author={Wei, Lai and Li, Yuting and Zheng, Kaipeng and Wang, Chen and Wang, Yue and Kong, Linghe and Sun, Lichao and Huang, Weiran},
journal={arXiv preprint arXiv:2505.22334},
year={2025}
}
``` |