Hugging Face's logo

MM-MVR
/
STAR-3B

STARMultiModalityConfig
Model card Files
xet
Community
MM-MVR commited on
Commit
c0409ce
·
verified ·
1 Parent(s): a2e0a64

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +325 -1
README.md CHANGED
@@ -2,4 +2,328 @@
2
  license: apache-2.0
3
  base_model:
4
  - Qwen/Qwen2.5-VL-3B-Instruct
5
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: apache-2.0
3
  base_model:
4
  - Qwen/Qwen2.5-VL-3B-Instruct
5
+ ---
6
+
7
+ <p align="center">
8
+ <img src="assets/star_logo.png" alt="STAR" width="560"/>
9
+ </p>
10
+
11
+ <p align="center">
12
+ <a href="https://arxiv.org/abs/xxxx.xxxxx">
13
+ <img
14
+ src="https://img.shields.io/badge/STAR-Paper-red?logo=arxiv&logoColor=red"
15
+ alt="STAR Paper on arXiv"
16
+ />
17
+ </a>
18
+ <a href="#">
19
+ <img
20
+ src="https://img.shields.io/badge/STAR-Project-0A66C2?logo=safari&logoColor=white"
21
+ alt="STAR Project"
22
+ />
23
+ </a>
24
+ <a href="#">
25
+ <img
26
+ src="https://img.shields.io/badge/STAR-Models-yellow?logo=huggingface&logoColor=yellow"
27
+ alt="STAR Models"
28
+ />
29
+ </a>
30
+ <a href="#">
31
+ <img
32
+ src="https://img.shields.io/badge/STAR-Demo-blue?logo=googleplay&logoColor=blue"
33
+ alt="STAR Demo"
34
+ />
35
+ </a>
36
+ <a href="#">
37
+ <img
38
+ src="https://img.shields.io/badge/STAR-Space-orange?logo=huggingface&logoColor=yellow"
39
+ alt="STAR HuggingFace Space"
40
+ />
41
+ </a>
42
+ </p>
43
+
44
+ # **STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning**
45
+
46
+
47
+ Welcome to the official repository for our paper: "STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning"
48
+
49
+
50
+ ## **Abstract**
51
+ Multimodal large language models (MLLMs) play a pivotal role in advancing the quest for general artificial intelligence. However, achieving unified target for multimodal understanding and generation remains challenging due to optimization conflicts and performance trade-offs. To effectively enhance generative performance while preserving existing comprehension capabilities, we introduce ***STAR***: *a **ST**acked **A**uto**R**egressive scheme for task-progressive unified multimodal learning*. This approach decomposes multimodal learning into multiple stages: understanding, generation, and editing. By freezing the parameters of the fundamental autoregressive (AR) model and progressively stacking isomorphic AR modules, it avoids cross-task interference while expanding the model's capabilities. Concurrently, we introduce a high-capacity VQ to enhance the granularity of image representations and employ an implicit reasoning mechanism to improve generation quality under complex conditions. Experiments demonstrate that STAR achieves state-of-the-art performance on GenEval (**0.91**), DPG-Bench (**87.44**), and ImgEdit (**4.34**), validating its efficacy for unified multimodal learning.
52
+
53
+ <div align="center">
54
+ <img src="assets/teaser.png" width=100%></img>
55
+ </div>
56
+
57
+ <div align="center">
58
+ <img src="assets/STAR-Framework.png" width=100%></img>
59
+ </div>
60
+
61
+
62
+
63
+
64
+ ## 📚 Preparation
65
+
66
+ ### Prepare the environment
67
+
68
+ 1. Set up environment
69
+ ```shell
70
+ git clone <repository-url>
71
+ cd STAR
72
+ conda create -n star python==3.11 -y
73
+ conda activate star
74
+ ```
75
+
76
+ 2. Install the required packages:
77
+ ```shell
78
+ # upgrade pip and setuptools if necessary
79
+ pip install -U pip setuptools
80
+
81
+ # install required packages
82
+ pip install -r requirements.txt
83
+
84
+ ```
85
+
86
+ ### Download Pre-trained Models
87
+ Download the necessary pre-trained models before proceeding to inference.
88
+
89
+ ```shell
90
+ STAR/checkpoints/STAR-7B.pt
91
+ STAR/checkpoints/VQ-Model.pt
92
+ ```
93
+
94
+ ### Configuration
95
+
96
+ The model configuration file `star/configs/STAR_Qwen2.5-VL-7B.json` contains all necessary parameters for model initialization. Make sure to update the paths in the configuration file to match your local setup.
97
+
98
+ ## 🔥 Quick Start
99
+
100
+ ### Demo
101
+
102
+ Run the interactive demo interface using Gradio.
103
+
104
+ ```shell
105
+ python3 gradio_app.py
106
+ ```
107
+
108
+ ### Inference
109
+
110
+ ### 1. Image Understanding
111
+
112
+ For visual question answering and image understanding tasks:
113
+
114
+ ```shell
115
+ python3 inference_understand.py \
116
+ --image-path "path/to/your/image.jpg" \
117
+ --question "What is in this image? Describe it in detail." \
118
+ --max-new-tokens 256 \
119
+ --model-config "star/configs/STAR_Qwen2.5-VL-7B.json" \
120
+ --checkpoint "checkpoints/STAR-7B.pt" \
121
+ --device "cuda:0"
122
+ ```
123
+
124
+ **Parameters:**
125
+ - `--image-path`: Path to the input image
126
+ - `--question`: Question or instruction for the model
127
+ - `--max-new-tokens`: Maximum number of tokens to generate (default: 256)
128
+ - `--model-config`: Path to model configuration file
129
+ - `--checkpoint`: Path to model checkpoint
130
+ - `--device`: Device to run inference on
131
+
132
+ ### 2. Text-to-Image Generation
133
+
134
+ For generating images from text prompts:
135
+
136
+ ```shell
137
+ python3 inference_generation.py \
138
+ --prompt "a photo of a cute cat" \
139
+ --save-path "./outputs/a photo of a cute cat.jpg" \
140
+ --num-images 1 \
141
+ --cfg 1.1 \
142
+ --topk 1000 \
143
+ --topp 0.8 \
144
+ --model-config "star/configs/STAR_Qwen2.5-VL-7B.json" \
145
+ --checkpoint "checkpoints/STAR-7B.pt" \
146
+ --diffusion-as-decoder \
147
+ --device "cuda:0"
148
+ ```
149
+
150
+ **Parameters:**
151
+ - `--prompt`: Text prompt for image generation
152
+ - `--save-path`: Path to save the generated image
153
+ - `--num-images`: Number of images to generate (default: 1)
154
+ - `--cfg`: Classifier-free guidance scale (default: 1.0)
155
+ - `--topk`: Top-k sampling parameter (default: 1000)
156
+ - `--topp`: Top-p sampling parameter (default: 0.8)
157
+ - `--diffusion-as-decoder`: Use diffusion model as decoder for high-quality generation
158
+
159
+ ### 3. Image Editing
160
+
161
+ For editing images based on text instructions:
162
+
163
+ ```shell
164
+ python3 inference_edit.py \
165
+ --image-path "./outputs/a photo of a cute cat.jpg" \
166
+ --instruction "change the color of cat to blue" \
167
+ --save-path "./outputs/edited_image.jpg" \
168
+ --cfg 1.1 \
169
+ --topk 1000 \
170
+ --topp 0.8 \
171
+ --model-config "star/configs/STAR_Qwen2.5-VL-7B.json" \
172
+ --checkpoint "checkpoints/STAR-7B.pt" \
173
+ --diffusion-as-decoder \
174
+ --device "cuda:0"
175
+ ```
176
+
177
+ **Parameters:**
178
+ - `--image-path`: Path to the input image to be edited
179
+ - `--instruction`: Text instruction describing the desired edit
180
+ - `--save-path`: Path to save the edited image
181
+ - `--cfg`: Classifier-free guidance scale for editing
182
+ - `--topk`: Top-k sampling parameter
183
+ - `--topp`: Top-p sampling parameter
184
+ - `--diffusion-as-decoder`: Use diffusion model for high-quality image decoding
185
+
186
+
187
+
188
+ ## 📊 Performance
189
+
190
+
191
+ ### 1. Visual Understanding
192
+
193
+ | Model | #LLM | MMB | MMStar | MathVista | SEED | MME-P | MMMU | OCRBench | POPE | DocVQA |
194
+ | ----- | ---- | --- | ------ | --------- | ---- | ----- | ---- | -------- | ---- | ------ |
195
+ | Janus-Pro | 7B | 79.2 | 87.4 | - | 72.1 | 1567.1 | 41.0 | - | - | - |
196
+ | BLIP3-o | 8B | 83.5 | - | - | 77.5 | 1682.6 | 50.6 | - | - | - |
197
+ | Show-o2 | 7B | 79.3 | 56.6 | - | 69.8 | 1620.0 | 48.9 | - | - | - |
198
+ | MetaQuery-XL | 7B | 83.5 | - | - | 76.9 | 1685.2 | 58.6 | - | - | - |
199
+ | Bagel | 14B | 85.0 | - | 73.1 | - | 1687.0 | 55.3 | - | - | - |
200
+ | Ovis-U1 | 1.5B | 77.8 | - | 69.4 | - | - | 51.1 | 88.3 | - | - |
201
+ | ILLUME+ | 3B | 80.8 | - | - | 73.3 | 1414.0 | 44.3 | 67.2 | 87.6 | 80.8 |
202
+ | X-Omni | 7B | 74.8 | - | - | 74.1 | - | - | 70.4 | 89.3 | 88.6 |
203
+ | **STAR-3B** | 3B | **80.1** | **55.8** | **62.3** | **74.0** | **1592.3** | **53.1** | **79.7** | **85.9** | **93.9** |
204
+ | **STAR-7B** | 7B | **83.9** | **63.9** | **68.1** | **77.0** | **1690.1** | **58.6** | **86.4** | **86.6** | **95.7** |
205
+
206
+ ### 2. Text-to-Image Generation
207
+
208
+ #### GenEval
209
+
210
+ | Model | Single | Two | Count. | Colors | Pos. | Color Attr. | Overall |
211
+ | ----- | ------ | --- | ------ | ------ | ---- | ----------- | ------- |
212
+ | <td colspan="6" align="center">**Generation-Only Models**</td> |
213
+ | SDXL | 0.98 | 0.74 | 0.39 | 0.85 | 0.15 | 0.23 | 0.55 |
214
+ | DALL-E | 0.96 | 0.87 | 0.47 | 0.83 | 0.43 | 0.45 | 0.67 |
215
+ | SD3-medium | 0.99 | 0.94 | 0.72 | 0.89 | 0.33 | 0.60 | 0.74 |
216
+ | FLUX.1-dev | 0.98 | 0.93 | 0.75 | 0.93 | 0.68 | 0.65 | 0.82 |
217
+ | OmniGen2 | 0.99 | 0.96 | 0.74 | 0.98 | 0.72 | 0.75 | 0.86 |
218
+ | <td colspan="6" align="center">**Unified Models**</td> |
219
+ | Emu3 | 0.99 | 0.81 | 0.42 | 0.80 | 0.49 | 0.45 | 0.66 |
220
+ | ILLUME+ | 0.99 | 0.88 | 0.62 | 0.84 | 0.42 | 0.53 | 0.72 |
221
+ | Janus-Pro | 0.99 | 0.89 | 0.59 | 0.90 | 0.79 | 0.66 | 0.80 |
222
+ | MetaQuery | - | - | - | - | - | - | 0.80 |
223
+ | BLIP3-o | - | - | - | - | - | - | 0.84 |
224
+ | UniWorld-V1 | 0.99 | 0.93 | 0.81 | 0.89 | 0.74 | 0.71 | 0.84 |
225
+ | Mogao | 1.00 | 0.97 | 0.83 | 0.93 | 0.84 | 0.80 | 0.89 |
226
+ | BAGEL | 0.98 | 0.95 | 0.84 | 0.95 | 0.78 | 0.77 | 0.88 |
227
+ | Show-o2 | 1.00 | 0.87 | 0.58 | 0.92 | 0.52 | 0.62 | 0.76 |
228
+ | GPT-4o | 0.99 | 0.92 | 0.85 | 0.92 | 0.75 | 0.61 | 0.84 |
229
+ | X-Omni | 0.98 | 0.95 | 0.75 | 0.91 | 0.71 | 0.68 | 0.83 |
230
+ | Ovis-U1 | 0.98 | 0.98 | 0.90 | 0.92 | 0.79 | 0.75 | 0.89 |
231
+ | **STAR-3B** | 0.98 | 0.87 | 0.85 | 0.91 | 0.79 | 0.76 | **0.86** |
232
+ | **STAR-7B** | 0.98 | 0.94 | 0.90 | 0.92 | 0.91 | 0.80 | **0.91** |
233
+
234
+ #### DPG-Bench
235
+
236
+ | Model | Global | Entity | Attr. | Relation | Other | Overall |
237
+ | ----- | ------ | ------ | ----- | -------- | ----- | ------- |
238
+ | <td colspan="5" align="center">**Generation-Only Models**</td> |
239
+ | SDXL | 83.27 | 82.43 | 80.91 | 86.76 | 80.41 | 74.65 |
240
+ | DALL-E | 90.97 | 89.61 | 88.39 | 90.58 | 89.83 | 83.50 |
241
+ | SD3-medium | 87.90 | 91.01 | 88.83 | 80.70 | 88.68 | 84.08 |
242
+ | FLUX.1-dev | 82.10 | 89.50 | 88.70 | 91.10 | 89.40 | 84.00 |
243
+ | OmniGen2 | 88.81 | 88.83 | 90.18 | 89.37 | 90.27 | 83.57 |
244
+ | <td colspan="5" align="center">**Unified Models**</td> |
245
+ | Emu3 | 85.21 | 86.68 | 86.84 | 90.22 | 83.15 | 80.60 |
246
+ | ILLUME+ | - | - | - | - | - | - |
247
+ | Janus-Pro | 86.90 | 88.90 | 89.40 | 89.32 | 89.48 | 84.19 |
248
+ | MetaQuery | - | - | - | - | - | 82.05 |
249
+ | BLIP3-o | - | - | - | - | - | 81.60 |
250
+ | UniWorld-V1 | 83.64 | 88.39 | 88.44 | 89.27 | 87.22 | 81.38 |
251
+ | Mogao | 82.37 | 90.03 | 88.26 | 93.18 | 85.40 | 84.33 |
252
+ | BAGEL | 88.94 | 90.37 | 91.29 | 90.82 | 88.67 | 85.07 |
253
+ | Show-o2 | 89.00 | 91.78 | 89.96 | 91.81 | 91.64 | 86.14 |
254
+ | GPT-4o | 82.27 | 91.27 | 87.67 | 93.85 | 88.71 | 86.23 |
255
+ | X-Omni | 84.80 | 92.59 | 90.63 | 94.75 | 84.20 | 87.65 |
256
+ | Ovis-U1 | 82.37 | 90.08 | 88.68 | 93.35 | 85.20 | 83.72 |
257
+ | **STAR-3B** | 93.00 | 90.49 | 91.71 | 90.72 | 92.75 | **87.30** |
258
+ | **STAR-7B** | 94.97 | 92.91 | 91.62 | 94.30 | 83.82 | **87.44** |
259
+
260
+ #### WISE (World Knowledge Reasoning)
261
+
262
+ | Model | Cultural | Time | Space | Biology | Physics | Chemistry | Overall |
263
+ | ----- | -------- | ---- | ----- | ------- | ------- | --------- | ------- |
264
+ | <td colspan="6" align="center">**Generation-Only Models**</td> |
265
+ | SD-XL | 0.43 | 0.48 | 0.47 | 0.44 | 0.45 | 0.27 | 0.43 |
266
+ | SD-3.5-large | 0.44 | 0.50 | 0.58 | 0.44 | 0.52 | 0.31 | 0.46 |
267
+ | FLUX.1-dev | 0.48 | 0.58 | 0.62 | 0.42 | 0.51 | 0.35 | 0.50 |
268
+ | <td colspan="6" align="center">**Unified Models**</td> |
269
+ | Emu3 | 0.34 | 0.45 | 0.48 | 0.41 | 0.45 | 0.27 | 0.39 |
270
+ | Janus-Pro-7B | 0.30 | 0.37 | 0.49 | 0.36 | 0.42 | 0.26 | 0.35 |
271
+ | MetaQuery-XL | 0.56 | 0.55 | 0.62 | 0.49 | 0.63 | 0.41 | 0.55 |
272
+ | BLIP3-o | - | - | - | - | - | - | 0.62 |
273
+ | BAGEL | 0.76 | 0.69 | 0.75 | 0.65 | 0.75 | 0.58 | 0.70 |
274
+ | GPT-4o | 0.94 | 0.64 | 0.98 | 0.93 | 0.98 | 0.95 | 0.89 |
275
+ | **STAR-3B** | 0.58 | 0.54 | 0.48 | 0.49 | 0.51 | 0.54 | **0.52** |
276
+ | **STAR-7B** | 0.61 | 0.67 | 0.61 | 0.74 | 0.69 | 0.66 | **0.66** |
277
+
278
+ ### 3. Image Editing
279
+
280
+ #### MagicBrush
281
+
282
+ | Model | L1 ↓ | CLIP-I ↑ | DINO ↑ |
283
+ | ----- | ---- | -------- | ------ |
284
+ | MagicBrush | 0.074 | 0.908 | 0.847 |
285
+ | Instruct-Pix2Pix | 0.114 | 0.851 | 0.744 |
286
+ | UltraEdit | 0.066 | 0.904 | 0.852 |
287
+ | ICEdit | 0.060 | 0.928 | 0.853 |
288
+ | OmniGen | 0.116 | 0.863 | 0.821 |
289
+ | UniReal | 0.081 | 0.903 | 0.837 |
290
+ | BAGEL | 0.074 | 0.914 | 0.827 |
291
+ | **STAR-3B** | **0.056** | **0.934** | **0.857** |
292
+ | **STAR-7B** | **0.060** | **0.931** | **0.853** |
293
+
294
+ #### ImgEdit-Bench
295
+
296
+ | Model | Add | Adjust | Extract | Replace | Remove | Background | Style | Hybrid | Action | Overall |
297
+ | ----- | --- | ------ | ------- | ------- | ------ | ---------- | ----- | ------ | ------ | ------- |
298
+ | <td colspan="9" align="center">**Editing-Only Models**</td> |
299
+ | MagicBrush | 2.84 | 1.58 | 1.51 | 1.97 | 1.58 | 1.75 | 2.38 | 1.62 | 1.22 | 1.90 |
300
+ | Instruct-Pix2Pix | 2.45 | 1.83 | 1.44 | 2.01 | 1.50 | 1.44 | 3.55 | 1.20 | 1.46 | 1.88 |
301
+ | AnyEdit | 3.18 | 2.95 | 1.88 | 2.47 | 2.23 | 2.24 | 2.85 | 1.56 | 2.65 | 2.45 |
302
+ | UltraEdit | 3.44 | 2.81 | 2.13 | 2.96 | 1.45 | 2.83 | 3.76 | 1.91 | 2.98 | 2.70 |
303
+ | Step1X-Edit | 3.88 | 3.14 | 1.76 | 3.40 | 2.41 | 3.16 | 4.63 | 2.64 | 2.52 | 3.06 |
304
+ | ICEdit | 3.58 | 3.39 | 1.73 | 3.15 | 2.93 | 3.08 | 3.84 | 2.04 | 3.68 | 3.05 |
305
+ | <td colspan="9" align="center">**Unified Models**</td> |
306
+ | GPT-4o | 4.61 | 4.33 | 2.90 | 4.35 | 3.66 | 4.57 | 4.93 | 3.96 | 4.89 | 4.20 |
307
+ | OmniGen | 3.47 | 3.04 | 1.71 | 2.94 | 2.43 | 3.21 | 4.19 | 2.24 | 3.38 | 2.96 |
308
+ | BAGEL | 3.56 | 3.31 | 1.70 | 3.30 | 2.62 | 3.24 | 4.49 | 2.38 | 4.17 | 3.20 |
309
+ | UniWorld-V1 | 3.82 | 3.64 | 2.27 | 3.47 | 3.24 | 2.99 | 4.21 | 2.96 | 2.74 | 3.26 |
310
+ | OmniGen2 | 3.57 | 3.06 | 1.77 | 3.74 | 3.20 | 3.57 | 4.81 | 2.52 | 4.68 | 3.44 |
311
+ | Ovis-U1 | 4.13 | 3.62 | 2.98 | 4.45 | 4.06 | 4.22 | 4.69 | 3.45 | 4.61 | 4.00 |
312
+ | **STAR-3B** | **4.26** | **4.06** | **3.78** | **4.46** | **4.34** | **4.19** | **4.53** | **3.29** | **4.38** | **4.14** |
313
+ | **STAR-7B** | **4.33** | **4.19** | **4.19** | **4.59** | **4.58** | **4.36** | **4.59** | **3.67** | **4.60** | **4.34** |
314
+
315
+
316
+ ## ✍️ Citation
317
+
318
+ ```bibtex
319
+ @article{2025star,
320
+ title = {STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning},
321
+ author = {Qin, Jie and Huang, Jiancheng and Qiao, Limeng and Ma, Lin},
322
+ journal = {arXiv preprint arXiv:},
323
+ year = {2025}
324
+ }
325
+ ```
326
+
327
+
328
+ ## 📜 License
329
+ STAR is licensed under the Apache 2.0.