Add pipeline tag and improve model card documentation
#1
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,123 +1,64 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
| 3 |
---
|
| 4 |
|
| 5 |
<div align="center">
|
| 6 |
|
| 7 |
-
<h1>
|
| 8 |
-
|
| 9 |
|
| 10 |
<h5 align="center">
|
| 11 |
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
|
| 16 |
-
Chunyi Peng<sup>1,3</sup>,
|
| 17 |
-
Zhipeng Xu<sup>1</sup>,
|
| 18 |
-
Zhenghao Liu<sup>1</sup>,
|
| 19 |
-
Yishan Li<sup>3</sup>,
|
| 20 |
-
Yukun Yan<sup>2</sup>,
|
| 21 |
-
Zhiyuan Liu<sup>2</sup>,
|
| 22 |
-
Yu Gu<sup>1</sup>
|
| 23 |
-
Minghe Yu<sup>1</sup>
|
| 24 |
-
Ge Yu<sup>1</sup>
|
| 25 |
-
Maosong Sun<sup>2</sup>
|
| 26 |
|
| 27 |
<sup>1</sup>Northeastern University, <sup>2</sup>Tsinghua University, <sup>3</sup>ModleBest Inc.
|
| 28 |
|
| 29 |
-
<h5 align="center"> If you find this project useful, please give us a star🌟.
|
| 30 |
</h5>
|
| 31 |
</div>
|
| 32 |
|
|
|
|
|
|
|
| 33 |
## News
|
| 34 |
-
|
|
|
|
| 35 |
|
| 36 |
-
## Environment
|
| 37 |
For training, answer generation, and evaluation processes:
|
| 38 |
```bash
|
| 39 |
conda create -n router python=3.11
|
| 40 |
conda activate router
|
| 41 |
pip install requirements_router.txt
|
| 42 |
```
|
| 43 |
-
For retriever and corpus construction processes:
|
| 44 |
-
```bash
|
| 45 |
-
conda create -n retriever python=3.11
|
| 46 |
-
conda activate retriever
|
| 47 |
-
pip install requirements_retriever.txt
|
| 48 |
-
```
|
| 49 |
-
|
| 50 |
-
## Corpora Construction
|
| 51 |
-
For the text corpus, you can download `enwiki-20241020` from [Huggingface](https://huggingface.co/datasets/hmhm1229/enwiki-20241020). Then preprocess, and index it with the following commands:
|
| 52 |
-
```bash
|
| 53 |
-
7z x enwiki-20241020-pages-articles-multistream.xml.zip.001
|
| 54 |
-
conda activate retriever
|
| 55 |
-
wikiextractor enwiki-20241020-pages-articles-multistream.xml.bz2 -o wiki_extracted
|
| 56 |
-
python wiki_preprocess.py
|
| 57 |
-
```
|
| 58 |
-
For the image corpus, you can directly download [M-BEIR](https://huggingface.co/datasets/TIGER-Lab/M-BEIR). To embed and index it you can follow the [repository](https://github.com/TIGER-AI-Lab/UniIR)
|
| 59 |
-
|
| 60 |
-
For the table corpus, you can download, embed and index Open-WikiTable following the [repository](https://github.com/sean0042/Open_WikiTable), or you can download directly the one we have already preprocessed from [here](https://huggingface.co/hmhm1229/table-retriever).
|
| 61 |
-
|
| 62 |
-
## Retrievers Preparation
|
| 63 |
-
For the Text-Image Retriever, you can directly download [UniIR](https://huggingface.co/TIGER-Lab/UniIR)
|
| 64 |
-
|
| 65 |
-
For the Table Retriever, you can train it with the help of [repository](https://github.com/sean0042/Open_WikiTable), or you can download it directly from [here](https://huggingface.co/hmhm1229/table-retriever).
|
| 66 |
|
| 67 |
-
## Datasets
|
| 68 |
-
We have prepared all the text datasets in `./datasets`, for images you need to download them from:
|
| 69 |
-
- `InfoSeek:` InfoSeek images can be downloaded from [OVEN](https://github.com/open-vision-language/oven/tree/main/image_downloads)
|
| 70 |
-
- `Dyn-VQA:` Dynamic VQA images can be downloaded from [DynVQA_en.202412](https://github.com/Alibaba-NLP/OmniSearch/blob/main/dataset/DynVQA_en/DynVQA_en.202412.jsonl)
|
| 71 |
-
- `WebQA:` WebQA images can be downloaded from [Google Drive](https://drive.google.com/drive/folders/19ApkbD5w0I5sV1IeQ9EofJRyAjKnA7tb)
|
| 72 |
-
|
| 73 |
-
## Training
|
| 74 |
-
If you do not want to train the model, you can download [R1-Router](https://huggingface.co/hmhm1229/R1-Router) and skip this section to [Evaluation](#evaluation)
|
| 75 |
-
### Data Synthesis
|
| 76 |
-
If you want to use the ready-to-use synthetic data directly, you can skip this section to [Step-GRPO Training](#step-grpo-training)
|
| 77 |
-
|
| 78 |
-
First, we need to synthesis the data step by step:
|
| 79 |
-
```bash
|
| 80 |
-
bash src/data_synthesis/data_synthesis.sh
|
| 81 |
-
```
|
| 82 |
-
### Step-GRPO Training
|
| 83 |
-
Our training framework is based on [EasyR1](https://github.com/hiyouga/EasyR1), only you need to do is to download it and replace some files with the files in `./Easy-R1`.
|
| 84 |
-
Then start training with the command:
|
| 85 |
-
```bash
|
| 86 |
-
conda activate router
|
| 87 |
-
bash examples/run_qwen2_5_vl_7b_stepgrpo.sh
|
| 88 |
-
```
|
| 89 |
## Evaluation
|
| 90 |
We provide the evaluation pipeline for the R1-Router:
|
| 91 |
```bash
|
| 92 |
bash evaluation.sh
|
| 93 |
```
|
| 94 |
-
|
| 95 |
```bash
|
| 96 |
conda activate router
|
| 97 |
cd src
|
| 98 |
python evaluate.py --dataset_name all --method "r1-router3"
|
| 99 |
```
|
| 100 |
|
| 101 |
-
##
|
| 102 |
-
|
| 103 |
-
- [EasyR1](https://github.com/hiyouga/EasyR1)
|
| 104 |
-
- [UniIR](https://huggingface.co/TIGER-Lab/UniIR)
|
| 105 |
-
- [Open-WikiTable](https://github.com/sean0042/Open_WikiTable)
|
| 106 |
-
- [OmniSearch](https://github.com/Alibaba-NLP/OmniSearch)
|
| 107 |
|
| 108 |
## Citation
|
| 109 |
-
|
| 110 |
-
```
|
| 111 |
-
@article{
|
| 112 |
-
title={
|
| 113 |
-
author={Peng, Chunyi and Xu, Zhipeng and Liu, Zhenghao and Li, Yishan and Yan, Yukun and Wang, Shuo and Liu, Zhiyuan and Gu, Yu and
|
| 114 |
-
year={2025}
|
| 115 |
-
url={https://arxiv.org/abs/2505.22095}
|
| 116 |
}
|
| 117 |
```
|
| 118 |
|
| 119 |
-
## Contact
|
| 120 |
-
If you have questions, suggestions,
|
| 121 |
-
```
|
| 122 |
-
hm.cypeng@gmail.com
|
| 123 |
-
```
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
pipeline_tag: text-retrieval
|
| 4 |
---
|
| 5 |
|
| 6 |
<div align="center">
|
| 7 |
|
| 8 |
+
<h1> MoRE: Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation </h1>
|
|
|
|
| 9 |
|
| 10 |
<h5 align="center">
|
| 11 |
|
| 12 |
+
[](https://arxiv.org/abs/2505.22095)
|
| 13 |
+
[](https://huggingface.co/hmhm1229/R1-Router-3B)
|
| 14 |
+
[](https://github.com/OpenBMB/R1-Router)
|
| 15 |
|
| 16 |
+
Chunyi Peng<sup>1,3</sup>, Zhipeng Xu<sup>1</sup>, Zhenghao Liu<sup>1</sup>, Yishan Li<sup>3</sup>, Yukun Yan<sup>2</sup>, Shuo Wang<sup>2</sup>, Zhiyuan Liu<sup>2</sup>, Yu Gu<sup>1</sup>, Minghe Yu<sup>1</sup>, Ge Yu<sup>1</sup>, Maosong Sun<sup>2</sup>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
<sup>1</sup>Northeastern University, <sup>2</sup>Tsinghua University, <sup>3</sup>ModleBest Inc.
|
| 19 |
|
|
|
|
| 20 |
</h5>
|
| 21 |
</div>
|
| 22 |
|
| 23 |
+
**R1-Router-3B** is a specialized routing model introduced as part of the **Mixture-of-Retrieval Experts (MoRE)** framework. MoRE enables Multimodal Large Language Models (MLLMs) to collaboratively interact with diverse retrieval experts (text, image, and table) for more effective knowledge exploitation. It learns to dynamically determine which expert to engage with based on the evolving reasoning state of the model.
|
| 24 |
+
|
| 25 |
## News
|
| 26 |
+
- **2026.04.03**: Our work is accepted by SIGIR2026! 🎉
|
| 27 |
+
- **2025.08.22**: We uploaded [R1-Router-3B](https://huggingface.co/hmhm1229/R1-Router-3B).
|
| 28 |
|
| 29 |
+
## Environment Setup
|
| 30 |
For training, answer generation, and evaluation processes:
|
| 31 |
```bash
|
| 32 |
conda create -n router python=3.11
|
| 33 |
conda activate router
|
| 34 |
pip install requirements_router.txt
|
| 35 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
## Evaluation
|
| 38 |
We provide the evaluation pipeline for the R1-Router:
|
| 39 |
```bash
|
| 40 |
bash evaluation.sh
|
| 41 |
```
|
| 42 |
+
Or you can evaluate specific results using:
|
| 43 |
```bash
|
| 44 |
conda activate router
|
| 45 |
cd src
|
| 46 |
python evaluate.py --dataset_name all --method "r1-router3"
|
| 47 |
```
|
| 48 |
|
| 49 |
+
## More Information
|
| 50 |
+
For detailed instructions on corpora construction, retriever preparation, and training using Stepwise Group Relative Policy Optimization (Step-GRPO), please refer to the official [GitHub repository](https://github.com/OpenBMB/R1-Router).
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
|
| 52 |
## Citation
|
| 53 |
+
If you find this project useful, please cite our paper:
|
| 54 |
+
```bibtex
|
| 55 |
+
@article{peng2025mixture,
|
| 56 |
+
title={Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation},
|
| 57 |
+
author={Peng, Chunyi and Xu, Zhipeng and Liu, Zhenghao and Li, Yishan and Yan, Yukun and Wang, Shuo and Liu, Zhiyuan and Gu, Yu and Minghe Yu and Ge Yu and Maosong Sun},
|
| 58 |
+
year={2025},
|
| 59 |
+
url={https://arxiv.org/abs/2505.22095}
|
| 60 |
}
|
| 61 |
```
|
| 62 |
|
| 63 |
+
## Contact
|
| 64 |
+
If you have questions, suggestions, or bug reports, please email: `hm.cypeng@gmail.com`.
|
|
|
|
|
|
|
|
|