README.md

---
base_model: Qwen/Qwen2-VL-7B-Instruct
library_name: transformers
license: apache-2.0
language:
- en
pipeline_tag: image-text-to-text
---

# UIPro: Unleashing Superior Interaction Capability For GUI Agents

<div align="center">

[\[💻Code\]](https://github.com/ZJULiHongxin/UIPro) [\[🚀Quick Start\]](#uses) [\[📝Paper\]](https://arxiv.org/abs/2509.17328)

</div>


![uipro_github_banner](https://cdn-uploads.huggingface.co/production/uploads/648e5a70df53671f33e94d52/VmLuH_usPK5hZOPPnYFhS.png)


## Model Details


![uipro_mainfigure](https://cdn-uploads.huggingface.co/production/uploads/648e5a70df53671f33e94d52/Kd5yOvqpFzoRlqEEL4KAS.png)

### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** Brave Group, CASIA
- **Model type:** Vision-Language Model
- **Language(s) (NLP):** English
- **License:** Apache License 2.0
- **Finetuned from model [optional]:** Qwen2-VL-7B-Instruct

### Model Sources [optional]

UIPro_1stage-7B is a GUI grounding model finetuned from Qwen2-VL-7B-Instruct.
<!-- Provide the basic links for the model. -->

- **Repository:** [https://github.com/ZJULiHongxin/UIPro](https://github.com/ZJULiHongxin/UIPro)
- **Paper [optional]:** [https://arxiv.org/abs/2509.17328](https://arxiv.org/abs/2509.17328)

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

First, ensure that the necessary dependencies are installed:
```
pip install transformers
pip install qwen-vl-utils
```

Inference code example:
```
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# Default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "HongxinLi/UIPro_1stage", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("HongxinLi/UIPro_1stage")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "./web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
            },

            # For ScreenSpot-v2, MOTIF, RefExp, and VisualWebBench Action Grounding
            {"type": "text", "text": "I want to {goal_info}. Please locate the target element I should interact with. (Output the center coordinates of the target)"},
            # For AutoGUI
            {"type": "text", "text": "Locate the element according to its detailed functionality description. {goal_info} (Output the center coordinates of the target)"},
            # For VisualWebBench Element Grounding
            {"type": "text", "text": "Locate the text "{goal_info}" (Output the center coordinates of the target)"},
        ],
    }
]
```


## Evaluation

It is recommended to use [AutoGUI evaluation suite](https://autogui-project.github.io/) based on [LMMS-EVAL](https://github.com/EvolvingLMMs-Lab/lmms-eval) to evaluate it on multiple GUI Grounding benchmarks.


### Results

| Model | Size | Input Res. | FuncGnd | ScreenSpot | ScreenSpot-v2 | MOTIF | RefExp | VWB EG | VWB AG |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| GPT-4o | - | AnyRes | 9.8 | 17.8 | 20.4 | 30.5 | 21.8 | 5.6 | 6.8 |
| Qwen2VL [1] | 72B | AnyRes | 47.7 | 71.4 | 73.2 | 80.3 | 77.7 | 60.5 | 62.1 |
| Qwen2VL [1] | 7B | AnyRes | 38.7 | 66.4 | 66.9 | 75.1 | 64.8 | 55.9 | 62.1 |
| CogAgent [2] | 18B | 1120 | 29.3 | 47.4 | 49.2 | 46.7 | 35.0 | 55.7 | 59.2 |
| SeeClick [3] | 10B | 448 | 19.8 | 53.4 | 54.0 | 11.1 | 58.1 | 39.2 | 27.2 |
| Ferret-UI [4] | 8B | AnyRes | 1.2 | 7.1 | 7.8 | 15.9 | 5.5 | 3.9 | 1.9 |
| UGround [5] | 7B | AnyRes | 48.8 | 74.8 | 76.5 | 72.4 | 73.6 | 85.2 | 63.1 |
| OS-ATLAS-Base [6] | 7B | AnyRes | 52.1 | 82.5 | 84.1 | 78.8 | 66.5 | 82.6 | 69.9 |
| **UIPro-Qwen2VL (ours)** | **7B** | **AnyRes** | **58.8** | **82.5** | **86.9** | **80.6** | **81.9** | **94.9** | **70.9** |
| Qwen2-VL [4] | 2B | AnyRes | 7.1 | 17.9 | 18.6 | 28.8 | 29.2 | 17.9 | 17.5 |
| **UIPro-SLiME (ours)** | **3B** | **AnyRes** | **58.3** | **60.7** | **61.1** | **73.3** | **59.0** | **60.0** | **40.8** |

**Comparison on the GUI element grounding benchmarks.** UIPro achieves impressive grounding accuracy, especially on FuncPred, RefExp, and VWB EG. AnyRes means using an image division strategy to handle images with variable resolutions.

References:
[1] [Qwen2VL](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)
[2] [CogAgent](https://huggingface.co/zai-org/cogagent-chat-hf)
[3] [SeeClick ](https://huggingface.co/cckevinn/SeeClick)
[4] [Ferret-UI](https://huggingface.co/jadechoghari/Ferret-UI-Llama8b)
[5] [UGround](https://huggingface.co/osunlp/UGround-V1-7B)
[6] [OS-ATLAS-Base](https://huggingface.co/OS-Copilot/OS-Atlas-Base-7B)



## Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**
```
@InProceedings{Li_2025_ICCV,
    author    = {Li, Hongxin and Su, Jingran and Chen, Jingfan and Ju, Zheng and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},
    title     = {UIPro: Unleashing Superior Interaction Capability For GUI Agents},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {1613-1623}
}
```

### Framework versions

- PEFT 0.11.1