Update README.md

README.md

@@ -0,0 +1,144 @@
+---
+base_model: Qwen/Qwen2-VL-7B-Instruct
+library_name: transformers
+license: apache-2.0
+language:
+- en
+pipeline_tag: image-text-to-text
+---
+
+# UIPro: Unleashing Superior Interaction Capability For GUI Agents
+
+<div align="center">
+
+[\[💻Code\]](https://github.com/ZJULiHongxin/UIPro) [\[🚀Quick Start\]](#uses) [\[📝Paper\]](https://arxiv.org/abs/2509.17328)
+
+</div>
+
+
+![uipro_github_banner](https://cdn-uploads.huggingface.co/production/uploads/648e5a70df53671f33e94d52/VmLuH_usPK5hZOPPnYFhS.png)
+
+
+## Model Details
+
+
+![uipro_mainfigure](https://cdn-uploads.huggingface.co/production/uploads/648e5a70df53671f33e94d52/Kd5yOvqpFzoRlqEEL4KAS.png)
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** Brave Group, CASIA
+- **Model type:** Vision-Language Model
+- **Language(s) (NLP):** English
+- **License:** Apache License 2.0
+- **Finetuned from model [optional]:** Qwen2-VL-7B-Instruct
+
+### Model Sources [optional]
+
+UIPro_1stage-7B is a GUI grounding model finetuned from Qwen2-VL-7B-Instruct.
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [https://github.com/ZJULiHongxin/UIPro](https://github.com/ZJULiHongxin/UIPro)
+- **Paper [optional]:** [https://arxiv.org/abs/2509.17328](https://arxiv.org/abs/2509.17328)
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+First, ensure that the necessary dependencies are installed:
+```
+pip install transformers
+pip install qwen-vl-utils
+```
+
+Inference code example:
+```
+from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
+from qwen_vl_utils import process_vision_info
+
+# Default: Load the model on the available device(s)
+model = Qwen2VLForConditionalGeneration.from_pretrained(
+    "HongxinLi/UIPro_1stage", torch_dtype="auto", device_map="auto"
+)
+processor = AutoProcessor.from_pretrained("HongxinLi/UIPro_1stage")
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": "./web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
+            },
+
+            # For ScreenSpot-v2, MOTIF, RefExp, and VisualWebBench Action Grounding
+            {"type": "text", "text": "I want to {goal_info}. Please locate the target element I should interact with. (Output the center coordinates of the target)"},
+            # For AutoGUI
+            {"type": "text", "text": "Locate the element according to its detailed functionality description. {goal_info} (Output the center coordinates of the target)"},
+            # For VisualWebBench Element Grounding
+            {"type": "text", "text": "Locate the text "{goal_info}" (Output the center coordinates of the target)"},
+        ],
+    }
+]
+```
+
+
+## Evaluation
+
+It is recommended to use [AutoGUI evaluation suite](https://autogui-project.github.io/) based on [LMMS-EVAL](https://github.com/EvolvingLMMs-Lab/lmms-eval) to evaluate it on multiple GUI Grounding benchmarks.
+
+
+### Results
+
+| Model | Size | Input Res. | FuncGnd | ScreenSpot | ScreenSpot-v2 | MOTIF | RefExp | VWB EG | VWB AG |
+| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
+| GPT-4o | - | AnyRes | 9.8 | 17.8 | 20.4 | 30.5 | 21.8 | 5.6 | 6.8 |
+| Qwen2VL [1] | 72B | AnyRes | 47.7 | 71.4 | 73.2 | 80.3 | 77.7 | 60.5 | 62.1 |
+| Qwen2VL [1] | 7B | AnyRes | 38.7 | 66.4 | 66.9 | 75.1 | 64.8 | 55.9 | 62.1 |
+| CogAgent [2] | 18B | 1120 | 29.3 | 47.4 | 49.2 | 46.7 | 35.0 | 55.7 | 59.2 |
+| SeeClick [3] | 10B | 448 | 19.8 | 53.4 | 54.0 | 11.1 | 58.1 | 39.2 | 27.2 |
+| Ferret-UI [4] | 8B | AnyRes | 1.2 | 7.1 | 7.8 | 15.9 | 5.5 | 3.9 | 1.9 |
+| UGround [5] | 7B | AnyRes | 48.8 | 74.8 | 76.5 | 72.4 | 73.6 | 85.2 | 63.1 |
+| OS-ATLAS-Base [6] | 7B | AnyRes | 52.1 | 82.5 | 84.1 | 78.8 | 66.5 | 82.6 | 69.9 |
+| **UIPro-Qwen2VL (ours)** | **7B** | **AnyRes** | **58.8** | **82.5** | **86.9** | **80.6** | **81.9** | **94.9** | **70.9** |
+| Qwen2-VL [4] | 2B | AnyRes | 7.1 | 17.9 | 18.6 | 28.8 | 29.2 | 17.9 | 17.5 |
+| **UIPro-SLiME (ours)** | **3B** | **AnyRes** | **58.3** | **60.7** | **61.1** | **73.3** | **59.0** | **60.0** | **40.8** |
+
+**Comparison on the GUI element grounding benchmarks.** UIPro achieves impressive grounding accuracy, especially on FuncPred, RefExp, and VWB EG. AnyRes means using an image division strategy to handle images with variable resolutions.
+
+References:
+[1] [Qwen2VL](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)
+[2] [CogAgent](https://huggingface.co/zai-org/cogagent-chat-hf)
+[3] [SeeClick ](https://huggingface.co/cckevinn/SeeClick)
+[4] [Ferret-UI](https://huggingface.co/jadechoghari/Ferret-UI-Llama8b)
+[5] [UGround](https://huggingface.co/osunlp/UGround-V1-7B)
+[6] [OS-ATLAS-Base](https://huggingface.co/OS-Copilot/OS-Atlas-Base-7B)
+
+
+
+## Citation
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+```
+@InProceedings{Li_2025_ICCV,
+    author    = {Li, Hongxin and Su, Jingran and Chen, Jingfan and Ju, Zheng and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},
+    title     = {UIPro: Unleashing Superior Interaction Capability For GUI Agents},
+    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
+    month     = {October},
+    year      = {2025},
+    pages     = {1613-1623}
+}
+```
+
+### Framework versions
+
+- PEFT 0.11.1