Improve model card

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +17 -47
README.md CHANGED
@@ -1,9 +1,10 @@
1
  ---
2
- license: apache-2.0
3
- language:
4
- - en
5
  base_model:
6
  - ByteDance-Seed/UI-TARS-1.5-7B
 
 
 
 
7
  pipeline_tag: image-text-to-text
8
  tags:
9
  - gui-agent
@@ -15,24 +16,23 @@ tags:
15
  - robustness
16
  - reinforcement-learning
17
  - grpo
18
- library_name: transformers
19
  ---
20
 
21
  # AgentHijack-Agent
22
 
23
- **AgentHijack-Agent** is the action-generation model released with the paper
24
- [*AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions*](https://AgentHijack.github.io) (ICML 2026).
25
 
26
  It is fine-tuned from [`UI-TARS-1.5-7B`](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) (Qwen2.5-VL architecture) using **Data-Augmented Group Relative Policy Optimization (DA-GRPO)** on the AgentHijack benchmark, with the goal of producing a computer-use agent that remains reliable under *common environment corruptions* (pop-ups, resolution changes, UI marks, subtitles, multi-apps, accidental touches, app minimization, network errors, and verification prompts).
27
 
28
  The same checkpoint serves a dual role in the AgentHijack-Agent framework:
29
-
30
  1. **Action generator** β€” produces the next GUI action from screenshots + history.
31
  2. **Onlooker** β€” summarizes behavioral changes between consecutive screenshots and performs an initial environment check before execution.
32
 
33
- - πŸ“„ **Paper:** *AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions* (ICML 2026)
34
- - 🌐 **Project page:** https://AgentHijack.github.io
35
- - 🧩 **Base model:** `ByteDance-Seed/UI-TARS-1.5-7B` (Qwen2.5-VL-7B architecture)
 
36
  - πŸ›οΈ **Affiliations:** TMLR Group, Hong Kong Baptist University
37
 
38
  ---
@@ -46,8 +46,6 @@ Compared with the base `UI-TARS-1.5-7B`, AgentHijack-Agent:
46
  - Recovers from **unexpected operations** (accidental touch, app minimization) via behavioral summarization.
47
  - Detects **environment errors** (network failure, login/verification prompts) up-front instead of looping on meaningless attempts.
48
 
49
- See Table 2 and Figure 8 of the paper for full results and qualitative trajectories.
50
-
51
  ---
52
 
53
  ## Model details
@@ -59,17 +57,14 @@ See Table 2 and Figure 8 of the paper for full results and qualitative trajector
59
  | Precision | `bfloat16` |
60
  | Context length | 128k tokens |
61
  | Image resolution | 1920 Γ— 1080 (native, paper default) |
62
- | Sharding | 4 Γ— `safetensors` shards |
63
  | Tokenizer | Inherited from UI-TARS-1.5-7B / Qwen2.5-VL |
64
 
65
  ### Training
66
 
67
- - **Algorithm:** Data-Augmented GRPO (DA-GRPO), an extension of GRPO that rolls out the same instruction across *different corrupted environments* drawn from a corruption set `C`, instead of a single clean environment.
68
  - **Framework:** [VERL](https://github.com/volcengine/verl).
69
- - **Data:** 128 tasks sampled from the AgentHijack benchmark (built on top of OSWorld with 9 configurable corruption types, 3,321 tasks total).
70
- - **Schedule:** 15 epochs.
71
- - **Reward:** `r = r_success + r_format`, with an experience-replay buffer (following ARPO) to mitigate sparse-reward batches.
72
- - **Optimization:** clip range [0.2, 0.3], KL loss disabled to encourage exploration.
73
 
74
  ---
75
 
@@ -93,48 +88,23 @@ wait()
93
  finished(content='xxx')
94
  ```
95
 
96
- ### Prompt template (action generator)
97
-
98
- ```
99
- You are a GUI agent. You are given a task and your action history, with
100
- screenshots. You need to perform the next action to complete the task.
101
-
102
- ## Output Format
103
- ```
104
- Thought: ...
105
- Action: ...
106
- ```
107
-
108
- ## Action Space
109
-
110
- {action_space}
111
-
112
- ## Note
113
- - Use {language} in `Thought` part.
114
- - Write a small plan and finally summarize your next action (with its target
115
- element) in one sentence in `Thought` part.
116
-
117
- ## User Instruction
118
- {instruction}
119
- ```
120
-
121
  ### Minimal inference example
122
 
123
  ```python
124
  from transformers import AutoProcessor, AutoModelForImageTextToText
125
  import torch
126
 
127
- model_id = "<your-username>/AgentHijack-Agent"
128
  processor = AutoProcessor.from_pretrained(model_id)
129
  model = AutoModelForImageTextToText.from_pretrained(
130
  model_id, torch_dtype=torch.bfloat16, device_map="auto"
131
  )
132
 
133
- # Build a chat with screenshot(s) + the action-generator prompt above,
134
  # then run model.generate(...) as usual.
135
  ```
136
 
137
- For the full agent framework (action generator + onlooker + environment checking), please refer to the code at [AgentHijack.github.io](https://AgentHijack.github.io).
138
 
139
  ---
140
 
@@ -156,4 +126,4 @@ If you use this model or the AgentHijack benchmark, please cite:
156
 
157
  ## Acknowledgements
158
 
159
- This model is built on top of [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) and the [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) family, with training infrastructure based on [VERL](https://github.com/volcengine/verl). The benchmark environment extends [OSWorld](https://os-world.github.io/).
 
1
  ---
 
 
 
2
  base_model:
3
  - ByteDance-Seed/UI-TARS-1.5-7B
4
+ language:
5
+ - en
6
+ library_name: transformers
7
+ license: apache-2.0
8
  pipeline_tag: image-text-to-text
9
  tags:
10
  - gui-agent
 
16
  - robustness
17
  - reinforcement-learning
18
  - grpo
 
19
  ---
20
 
21
  # AgentHijack-Agent
22
 
23
+ **AgentHijack-Agent** is the action-generation model released with the paper
24
+ [*AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions*](https://huggingface.co/papers/2605.25707) (ICML 2026).
25
 
26
  It is fine-tuned from [`UI-TARS-1.5-7B`](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) (Qwen2.5-VL architecture) using **Data-Augmented Group Relative Policy Optimization (DA-GRPO)** on the AgentHijack benchmark, with the goal of producing a computer-use agent that remains reliable under *common environment corruptions* (pop-ups, resolution changes, UI marks, subtitles, multi-apps, accidental touches, app minimization, network errors, and verification prompts).
27
 
28
  The same checkpoint serves a dual role in the AgentHijack-Agent framework:
 
29
  1. **Action generator** β€” produces the next GUI action from screenshots + history.
30
  2. **Onlooker** β€” summarizes behavioral changes between consecutive screenshots and performs an initial environment check before execution.
31
 
32
+ - πŸ“„ **Paper:** [AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions](https://huggingface.co/papers/2605.25707)
33
+ - 🌐 **Project page:** [https://AgentHijack.github.io](https://AgentHijack.github.io)
34
+ - πŸ’» **Code:** [https://github.com/tmlr-group/AgentHijack](https://github.com/tmlr-group/AgentHijack)
35
+ - πŸ‘₯ **Authors:** Jingwei Sun, Jianing Zhu, Yuanyi Li, Tongliang Liu, Xia Hu, and Bo Han
36
  - πŸ›οΈ **Affiliations:** TMLR Group, Hong Kong Baptist University
37
 
38
  ---
 
46
  - Recovers from **unexpected operations** (accidental touch, app minimization) via behavioral summarization.
47
  - Detects **environment errors** (network failure, login/verification prompts) up-front instead of looping on meaningless attempts.
48
 
 
 
49
  ---
50
 
51
  ## Model details
 
57
  | Precision | `bfloat16` |
58
  | Context length | 128k tokens |
59
  | Image resolution | 1920 Γ— 1080 (native, paper default) |
 
60
  | Tokenizer | Inherited from UI-TARS-1.5-7B / Qwen2.5-VL |
61
 
62
  ### Training
63
 
64
+ - **Algorithm:** Data-Augmented GRPO (DA-GRPO), an extension of GRPO that rolls out the same instruction across *different corrupted environments* drawn from a corruption set `C`.
65
  - **Framework:** [VERL](https://github.com/volcengine/verl).
66
+ - **Data:** 128 tasks sampled from the AgentHijack benchmark (built on top of OSWorld).
67
+ - **Reward:** `r = r_success + r_format`, with an experience-replay buffer to mitigate sparse-reward batches.
 
 
68
 
69
  ---
70
 
 
88
  finished(content='xxx')
89
  ```
90
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
91
  ### Minimal inference example
92
 
93
  ```python
94
  from transformers import AutoProcessor, AutoModelForImageTextToText
95
  import torch
96
 
97
+ model_id = "TMLR-Group-HF/AgentHijack-Agent"
98
  processor = AutoProcessor.from_pretrained(model_id)
99
  model = AutoModelForImageTextToText.from_pretrained(
100
  model_id, torch_dtype=torch.bfloat16, device_map="auto"
101
  )
102
 
103
+ # Build a chat with screenshot(s) + the action-generator prompt,
104
  # then run model.generate(...) as usual.
105
  ```
106
 
107
+ For the full agent framework (action generator + onlooker + environment checking), please refer to the code at [GitHub](https://github.com/tmlr-group/AgentHijack).
108
 
109
  ---
110
 
 
126
 
127
  ## Acknowledgements
128
 
129
+ This model is built on top of [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) and the [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) family, with training infrastructure based on [VERL](https://github.com/volcengine/verl). The benchmark environment extends [OSWorld](https://os-world.github.io/).