|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- Qwen/Qwen2.5-VL-3B-Instruct |
|
|
pipeline_tag: visual-question-answering |
|
|
--- |
|
|
|
|
|
|
|
|
This repository contains the model presented in [UI-R1: Enhancing Action Prediction of GUI Agents by Reinforcement Learning](https://huggingface.co/papers/2503.21620). |
|
|
|
|
|
Project page: https://github.com/lll6gg/UI-R1 |
|
|
|
|
|
New version: [UI-R1-E-3B](https://huggingface.co/LZXzju/Qwen2.5-VL-3B-UI-R1-E) |
|
|
|
|
|
## Benchmark 1: ScreenSpotV2 |
|
|
|
|
|
| ScreenSpotV2 | inference mode | Mobile-T | Mobile-I | Desktop-T | Desktop-I | Web-T | Web-I | Avg↑ / Len↓ | |
|
|
| ------------- | -------------- | -------- | -------- | --------- | --------- | -------- | -------- | ----------------- | |
|
|
| OS-ATLAS-7B | w/o thinking | 95.2 | 75.8 | 90.7 | 63.6 | 90.6 | 77.3 | 84.1 / | |
|
|
| UI-TARS-7B | w/o thinking | 95.2 | 79.1 | 90.7 | 68.6 | 90.6 | 78.3 | 84.7 / | |
|
|
| UI-R1-3B (v1) | w/ thinking | 96.2 | **84.3** | 92.3 | 63.6 | 89.2 | 75.4 | 85.4 / 67 | |
|
|
| GUI-R1-3B | w/ thinking | 97.6 | 78.2 | 94.3 | 64.3 | 91.0 | 72.4 | 85.0 / 80 | |
|
|
| UI-R1-3B (v2) | w/ thinking | 97.6 | 79.6 | 92.3 | 67.9 | 88.9 | 77.8 | 85.8 / 60 | |
|
|
| **UI-R1-E-3B** | w/o thinking | **98.2** | 83.9 | **94.8** | **75.0** | **93.2** | **83.7** | **89.5** / **28** | |
|
|
## Benchmark 2: ScreenSpot-Pro |
|
|
|
|
|
| ScreenSpot-Pro | inference mode | Average Length↓ | Average Accuracy↑ | |
|
|
| -------------- | -------------- | --------------- | ---------------- | |
|
|
| UGround-7B | w/o thinking | - | 16.5 | |
|
|
| OS-ATLAS-7B | w/o thinking | - | 18.9 | |
|
|
| UI-R1-3B (v1) | w/ thinking | 102 | 17.8 | |
|
|
| GUI-R1-3B | w/ thinking | 114 | 26.6 | |
|
|
| UI-R1-3B (v2) | w/ thinking | 129 | 29.8 | |
|
|
| **UI-R1-E-3B** | w/o thinking | **28** | **33.5** | |
|
|
## Leaderboard: UI-I2E-Bench |
|
|
| Model | ScreenSpot | UI-I2E-Bench Avg | ScreenSpot-Pro | Avg | |
|
|
| :------------: | :--------: | :--------------: | :------------: | :--: | |
|
|
| UI-TARS-1.5-7B | 88.1 | 73.2 | 42.2 | 67.8 | |
|
|
| Uground-V1-72B | 89.7 | 76.3 | 34.3 | 66.8 | |
|
|
| UI-TARS-72B | 88.4 | 73.7 | 38.1 | 66.7 | |
|
|
| **UI-R1-E-3B** | 89.2 | 69.1 | 33.5 | 63.9 | |
|
|
| Uground-V1-7B | 87.1 | 70.3 | 31.1 | 62.8 | |
|
|
| InfiGUI-R1 | 87.5 | 69.7 | 29.6 | 62.3 | |
|
|
| UI-TARS-7B | 89.5 | 61.4 | 35.7 | 62.2 | |
|
|
| Qwen2.5-VL-72B | 87.1 | 51.4 | 43.6 | 60.7 | |
|
|
| UI-I2E-VLM-7B | 82.5 | 69.5 | 23.6 | 58.5 | |
|
|
| UI-TARS-2B | 82.3 | 62 | 27.7 | 57.3 | |
|
|
| Qwen2.5-VL-7B | 84.7 | 53.8 | 29 | 55.8 | |
|
|
| OmniParser-V2 | 72 | 54.8 | 39.6 | 55.5 | |
|
|
| Uground-V1-2B | 78.8 | 57.4 | 26.6 | 54.3 | |
|
|
| OS-Atlas-7B | 82.5 | 58.6 | 18.9 | 53.3 | |
|
|
| **UI-R1-3B** | 83.3 | 58.5 | 17.8 | 53.2 | |
|
|
| UGround-7B | 74.1 | 54.2 | 16.5 | 48.3 | |
|
|
| UI-I2E-VLM-4B | 70.4 | 53.4 | 12.2 | 45.3 | |
|
|
| OmniParser | 73.9 | 53.1 | 8.3 | 45.1 | |
|
|
| ShowUI-2B | 76.8 | 41.5 | 7.7 | 42 | |
|
|
| Qwen2.5-VL-3B | 55.5 | 41.7 | 23.9 | 41.3 | |
|
|
| Aguvis-7B | 84.4 | 53.2 | 22.9 | 40.4 | |
|
|
| OS-Atlas-4B | 70.1 | 44.3 | 3.7 | 39.4 | |
|
|
| Qwen2-VL-7B | 42.6 | 48.7 | 1.6 | 31 | |
|
|
| Seeclick | 55.8 | 26.4 | 1.1 | 27.8 | |
|
|
| InternVL2-4B | 4.2 | 0.9 | 0.3 | 1.8 | |
|
|
|
|
|
## Evaluation Code for GUI Grounding |
|
|
|
|
|
1. Generation for UI-R1-E-3B： |
|
|
|
|
|
```python |
|
|
model = Qwen2_5_VLForConditionalGeneration.from_pretrained( |
|
|
args.model_path, |
|
|
torch_dtype=torch.bfloat16, |
|
|
attn_implementation="flash_attention_2", |
|
|
device_map="cpu", |
|
|
) |
|
|
model = model.to(torch.device(rank)) |
|
|
model = model.eval() |
|
|
processor = AutoProcessor.from_pretrained(ori_processor_path) |
|
|
question_template = ( |
|
|
f"In this UI screenshot, I want to perform the command '{task_prompt}'.\n" |
|
|
"Please provide the action to perform (enumerate in ['click', 'scroll']) and the coordinate where the cursor is moved to(integer) if click is performed.\n" |
|
|
"Output the thinking process in <think> </think> and final answer in <answer> </answer> tags." |
|
|
"The output answer format should be as follows:\n" |
|
|
"<think> ... </think> <answer>[{'action': enum['click', 'scroll'], 'coordinate': [x, y]}]</answer>\n" |
|
|
"Please strictly follow the format." |
|
|
) |
|
|
query = '<image>\n' + question_template |
|
|
messages = [ |
|
|
{ |
|
|
"role": "user", |
|
|
"content": [ |
|
|
{"type": "image", "image": image_path} |
|
|
] + [{"type": "text", "text": query}], |
|
|
} |
|
|
] |
|
|
text = processor.apply_chat_template( |
|
|
messages, tokenize=False, add_generation_prompt=True |
|
|
) |
|
|
image_inputs, video_inputs = process_vision_info(messages) |
|
|
inputs = processor( |
|
|
text=[text], |
|
|
images=image_inputs, |
|
|
videos=video_inputs, |
|
|
padding=True, |
|
|
return_tensors="pt", |
|
|
) |
|
|
generated_ids = model.generate(**inputs, max_new_tokens=1024) |
|
|
generated_ids_trimmed = [ |
|
|
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) |
|
|
] |
|
|
response = processor.batch_decode( |
|
|
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False |
|
|
) |
|
|
response = response[0] |
|
|
pred_coord, _ = extract_coord(response) |
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
2. Rescale the predicted coordinate according to the image resize (especially image_size > 12845056) |
|
|
|
|
|
```python |
|
|
image = Image.open(image_path) |
|
|
origin_width, origin_height = image.size |
|
|
resized_height,resized_width = smart_resize(origin_height,origin_width,max_pixels=12845056) |
|
|
scale_x = origin_width / resized_width |
|
|
scale_y = origin_height / resized_height |
|
|
pred_coord[0] = int(pred_coord[0] * scale_x) |
|
|
pred_coord[1] = int(pred_coord[1] * scale_y) |
|
|
``` |
|
|
|
|
|
Function smart_resize is from Qwen2VL： |
|
|
|
|
|
```python |
|
|
import math |
|
|
def smart_resize( |
|
|
height: int, width: int, factor: int = 28, min_pixels: int = 56 * 56, max_pixels: int = 14 * 14 * 4 * 1280 |
|
|
): |
|
|
"""Rescales the image so that the following conditions are met: |
|
|
|
|
|
1. Both dimensions (height and width) are divisible by 'factor'. |
|
|
|
|
|
2. The total number of pixels is within the range ['min_pixels', 'max_pixels']. |
|
|
|
|
|
3. The aspect ratio of the image is maintained as closely as possible. |
|
|
|
|
|
""" |
|
|
if height < factor or width < factor: |
|
|
raise ValueError(f"height:{height} or width:{width} must be larger than factor:{factor}") |
|
|
elif max(height, width) / min(height, width) > 200: |
|
|
raise ValueError( |
|
|
f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}" |
|
|
) |
|
|
h_bar = round(height / factor) * factor |
|
|
w_bar = round(width / factor) * factor |
|
|
if h_bar * w_bar > max_pixels: |
|
|
beta = math.sqrt((height * width) / max_pixels) |
|
|
h_bar = math.floor(height / beta / factor) * factor |
|
|
w_bar = math.floor(width / beta / factor) * factor |
|
|
elif h_bar * w_bar < min_pixels: |
|
|
beta = math.sqrt(min_pixels / (height * width)) |
|
|
h_bar = math.ceil(height * beta / factor) * factor |
|
|
w_bar = math.ceil(width * beta / factor) * factor |
|
|
return h_bar, w_bar |
|
|
``` |
|
|
|
|
|
|
