Improve model card: Add pipeline tag, abstract, and project resources

README.md

@@ -1,8 +1,31 @@
 ---
 library_name: transformers
-tags: []
+pipeline_tag: image-text-to-text
+license: apache-2.0
+tags:
+- gui
+- agent
+- visual-grounding
+- multimodal
+- reinforcement-learning
+- qwen2.5
 ---
 
+# GTA1: GUI Test-time Scaling Agent
+
+This repository contains the GUI grounding model presented in the paper [GTA1: GUI Test-time Scaling Agent](https://huggingface.co/papers/2507.05791).
+
+## Abstract
+
+Graphical user interface (GUI) agents autonomously operate across platforms (e.g., Linux) to complete tasks by interacting with visual elements. Specifically, a user instruction is decomposed into a sequence of action proposals, each corresponding to an interaction with the GUI. After each action, the agent observes the updated GUI environment to plan the next step. However, two main challenges arise: i) resolving ambiguity in task planning (i.e., the action proposal sequence), where selecting an appropriate plan is non-trivial, as many valid ones may exist; ii) accurately grounding actions in complex and high-resolution interfaces, i.e., precisely interacting with visual targets. This paper investigates the two aforementioned challenges with our GUI Test-time Scaling Agent, namely GTA1. First, to select the most appropriate action proposal, we introduce a test-time scaling method. At each step, we sample multiple candidate action proposals and leverage a judge model to evaluate and select the most suitable one. It trades off computation for better decision quality by concurrent sampling, shortening task execution steps, and improving overall performance. Second, we propose a model that achieves improved accuracy when grounding the selected action proposal to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates visual grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, our method establishes state-of-the-art performance across diverse benchmarks. For example, GTA1-7B achieves 50.1%, 92.4%, and 67.7% accuracies on Screenspot-Pro, Screenspot-V2, and OSWorld-G, respectively. When paired with a planner applying our test-time scaling strategy, it exhibits state-of-the-art agentic performance (e.g., 45.2% task success rate on OSWorld). We open-source our code and models here.
+
+## Project Resources
+
+*   **Paper:** [GTA1: GUI Test-time Scaling Agent](https://huggingface.co/papers/2507.05791)
+*   **GitHub Repository:** [Yan98/GTA1](https://github.com/Yan98/GTA1)
+*   **Project Page:** [yan98.github.io/GTA1](https://yan98.github.io/GTA1/)
+*   **Blog Post:** [Grounding R1](https://huggingface.co/blog/HelloKKMe/grounding-r1)
+
 # Introduction
 
 Reinforcement learning (RL) (e.g., GRPO) helps with grounding because of its inherent objective alignment—rewarding successful clicks—rather than encouraging long textual Chain-of-Thought (CoT) reasoning. Unlike approaches that rely heavily on verbose CoT reasoning, GRPO directly incentivizes actionable and grounded responses. Based on findings from our [blog](https://huggingface.co/blog/HelloKKMe/grounding-r1), we share state-of-the-art GUI grounding models trained using GRPO.
@@ -11,27 +34,26 @@ Reinforcement learning (RL) (e.g., GRPO) helps with grounding because of its inh
 
 We follow the standard evaluation protocol and benchmark our model on three challenging datasets. Our method consistently achieves the best results among all open-source model families. Below are the comparative results:
 
-| **Model**         | **Size** | **Open Source** | **ScreenSpot-V2** | **ScreenSpotPro** | **OSWORLD-G** |
-|-------------------|:--------:|:---------------:|:-----------------:|:-----------------:|:-----------------:|
-| OpenAI CUA        | —        | ❌              | 87.9              | 23.4              |        —          |
-| Claude 3.7        | —        | ❌              | 87.6              | 27.7              |        —          | 
-| JEDI-7B           | 7B       | ✅              | 91.7              | 39.5              | 54.1              |
-| SE-GUI            | 7B       | ✅              | 90.3              | 47.0              |        —          |
-| UI-TARS           | 7B       | ✅              | 91.6              | 35.7              | 47.5              |
-| UI-TARS-1.5*       | 7B       | ✅              | 89.7*                 | 42.0*              |  64.2* | 
-| UGround-v1-7B     | 7B       | ✅              |  —                | 31.1              |   36.4        |
-| Qwen2.5-VL-32B-Instruct | 32B | ✅              |  91.9*                | 48.0              |        59.6*      |    |
-| UGround-v1-72B    | 72B      | ✅              |  —                | 34.5              |        —          |
-| Qwen2.5-VL-72B-Instruct | 72B | ✅              |  94.00*                | 53.3              |        62.2*          |
-| UI-TARS           | 72B      | ✅              | 90.3              | 38.1              |        —          |
-| GTA1 (Ours)              | 7B       | ✅              | 92.4 <sub>*(∆ +2.7)*</sub>             | 50.1<sub>*(∆ +8.1)*</sub>              | 67.7 <sub>*(∆ +3.5)*</sub>              |
-| GTA1 (Ours)              | 32B      | ✅              | 93.2 <sub>*(∆ +1.3)*</sub>             | 53.6 <sub>*(∆ +5.6)*</sub>             |        61.9<sub>*(∆ +2.3)*</sub>          |
-| GTA1 (Ours)              | 72B      | ✅              | 94.8<sub>*(∆ +0.8)*</sub>              | 58.4 <sub>*(∆ +5.1)*</sub>             |        66.7<sub>*(∆ +4.5)*</sub>          |
-
-
-> **Note:**  
-> - Model size is indicated in billions (B) of parameters.  
-> - A dash (—) denotes results that are currently unavailable.  
+| **Model** | **Size** | **Open Source** | **ScreenSpot-V2** | **ScreenSpotPro** | **OSWORLD-G** |
+|---|:---:|:---:|:---:|:---:|:---:|
+| OpenAI CUA | — | ❌ | 87.9 | 23.4 | — |
+| Claude 3.7 | — | ❌ | 87.6 | 27.7 | — |
+| JEDI-7B | 7B | ✅ | 91.7 | 39.5 | 54.1 |
+| SE-GUI | 7B | ✅ | 90.3 | 47.0 | — |
+| UI-TARS | 7B | ✅ | 91.6 | 35.7 | 47.5 |
+| UI-TARS-1.5* | 7B | ✅ | 89.7* | 42.0* | 64.2* |
+| UGround-v1-7B | 7B | ✅ | — | 31.1 | 36.4 |
+| Qwen2.5-VL-32B-Instruct | 32B | ✅ | 91.9* | 48.0 | 59.6* |
+| UGround-v1-72B | 72B | ✅ | — | 34.5 | — |
+| Qwen2.5-VL-72B-Instruct | 72B | ✅ | 94.00* | 53.3 | 62.2* |
+| UI-TARS | 72B | ✅ | 90.3 | 38.1 | — |
+| GTA1 (Ours) | 7B | ✅ | 92.4 <sub>*(∆ +2.7)*</sub> | 50.1<sub>*(∆ +8.1)*</sub> | 67.7 <sub>*(∆ +3.5)*</sub> |
+| GTA1 (Ours) | 32B | ✅ | 93.2 <sub>*(∆ +1.3)*</sub> | 53.6 <sub>*(∆ +5.6)*</sub> | 61.9<sub>*(∆ +2.3)*</sub> |
+| GTA1 (Ours) | 72B | ✅ | 94.8<sub>*(∆ +0.8)*</sub> | 58.4 <sub>*(∆ +5.1)*</sub> | 66.7<sub>*(∆ +4.5)*</sub> |
+
+> **Note:**
+> - Model size is indicated in billions (B) of parameters.
+> - A dash (—) denotes results that are currently unavailable.
 > - A superscript asterisk (﹡) denotes our evaluated result.
 > - UI-TARS-1.5 7B, Qwen2.5-VL-32B-Instruct, and Qwen2.5-VL-72B-Instruct are applied as our baseline models.
 > - ∆ indicates the performance improvement (∆) of our model compared to its baseline.
@@ -119,10 +141,36 @@ generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(in
 output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)[0]
 
 # Extract and rescale coordinates
-pred_x, pred_y  = extract_coordinates(output_text) 
+pred_x, pred_y  = extract_coordinates(output_text)
 pred_x*=scale_x
-pred_y*=scale_y 
+pred_y*=scale_y
 print(pred_x,pred_y)
 ```
 
-Refer to our [code](https://github.com/Yan98/GTA1) for more details.
+Refer to our [code](https://github.com/Yan98/GTA1) for more details.
+
+## Agent Performance
+
+Refer to an inference example [here](https://github.com/xlang-ai/OSWorld/pull/246/files#diff-2b758e4fafd9a52ee08bd6072f64297e4d880193fcf3f0e480da954a6711afa7).
+
+## Contact
+
+Please contact `yan.yang@anu.edu.au` for any queries.
+
+## Acknowledgement
+
+We thank the open-source projects: [VLM-R1](https://github.com/om-ai-lab/VLM-R1), [Jedi](https://github.com/xlang-ai/OSWorld/blob/main/mm_agents/jedi_7b_agent.py), and [Agent-S2](https://github.com/simular-ai/Agent-S).
+
+## Citation
+If you use this repository or find it helpful in your research, please cite it as follows:
+```bibtex
+@misc{yang2025gta1guitesttimescaling,
+      title={GTA1: GUI Test-time Scaling Agent},
+      author={Yan Yang and Dongxu Li and Yutong Dai and Yuhao Yang and Ziyang Luo and Zirui Zhao and Zhiyuan Hu and Junzhe Huang and Amrita Saha and Zeyuan Chen and Ran Xu and Liyuan Pan and Caiming Xiong and Junnan Li},
+      year={2025},
+      eprint={2507.05791},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI},
+      url={https://arxiv.org/abs/2507.05791},
+}
+```