nielsr HF Staff commited on
Commit
ee92da4
Β·
verified Β·
1 Parent(s): 4025119

Improve model card

Browse files

Hi! I'm Niels, part of the community science team at Hugging Face.

This PR improves the model card for **AgentHijack-Agent**. The updates include:
- Linking the model to its research paper on the Hugging Face papers page.
- Adding a direct link to the GitHub repository for easier access to the code.
- Listing the full author team.
- Formatting the metadata and content for better discoverability and readability.

These changes help researchers and users find, cite, and implement your work more effectively!

Files changed (1) hide show
  1. README.md +17 -47
README.md CHANGED
@@ -1,9 +1,10 @@
1
  ---
2
- license: apache-2.0
3
- language:
4
- - en
5
  base_model:
6
  - ByteDance-Seed/UI-TARS-1.5-7B
 
 
 
 
7
  pipeline_tag: image-text-to-text
8
  tags:
9
  - gui-agent
@@ -15,24 +16,23 @@ tags:
15
  - robustness
16
  - reinforcement-learning
17
  - grpo
18
- library_name: transformers
19
  ---
20
 
21
  # AgentHijack-Agent
22
 
23
- **AgentHijack-Agent** is the action-generation model released with the paper
24
- [*AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions*](https://AgentHijack.github.io) (ICML 2026).
25
 
26
  It is fine-tuned from [`UI-TARS-1.5-7B`](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) (Qwen2.5-VL architecture) using **Data-Augmented Group Relative Policy Optimization (DA-GRPO)** on the AgentHijack benchmark, with the goal of producing a computer-use agent that remains reliable under *common environment corruptions* (pop-ups, resolution changes, UI marks, subtitles, multi-apps, accidental touches, app minimization, network errors, and verification prompts).
27
 
28
  The same checkpoint serves a dual role in the AgentHijack-Agent framework:
29
-
30
  1. **Action generator** β€” produces the next GUI action from screenshots + history.
31
  2. **Onlooker** β€” summarizes behavioral changes between consecutive screenshots and performs an initial environment check before execution.
32
 
33
- - πŸ“„ **Paper:** *AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions* (ICML 2026)
34
- - 🌐 **Project page:** https://AgentHijack.github.io
35
- - 🧩 **Base model:** `ByteDance-Seed/UI-TARS-1.5-7B` (Qwen2.5-VL-7B architecture)
 
36
  - πŸ›οΈ **Affiliations:** TMLR Group, Hong Kong Baptist University
37
 
38
  ---
@@ -46,8 +46,6 @@ Compared with the base `UI-TARS-1.5-7B`, AgentHijack-Agent:
46
  - Recovers from **unexpected operations** (accidental touch, app minimization) via behavioral summarization.
47
  - Detects **environment errors** (network failure, login/verification prompts) up-front instead of looping on meaningless attempts.
48
 
49
- See Table 2 and Figure 8 of the paper for full results and qualitative trajectories.
50
-
51
  ---
52
 
53
  ## Model details
@@ -59,17 +57,14 @@ See Table 2 and Figure 8 of the paper for full results and qualitative trajector
59
  | Precision | `bfloat16` |
60
  | Context length | 128k tokens |
61
  | Image resolution | 1920 Γ— 1080 (native, paper default) |
62
- | Sharding | 4 Γ— `safetensors` shards |
63
  | Tokenizer | Inherited from UI-TARS-1.5-7B / Qwen2.5-VL |
64
 
65
  ### Training
66
 
67
- - **Algorithm:** Data-Augmented GRPO (DA-GRPO), an extension of GRPO that rolls out the same instruction across *different corrupted environments* drawn from a corruption set `C`, instead of a single clean environment.
68
  - **Framework:** [VERL](https://github.com/volcengine/verl).
69
- - **Data:** 128 tasks sampled from the AgentHijack benchmark (built on top of OSWorld with 9 configurable corruption types, 3,321 tasks total).
70
- - **Schedule:** 15 epochs.
71
- - **Reward:** `r = r_success + r_format`, with an experience-replay buffer (following ARPO) to mitigate sparse-reward batches.
72
- - **Optimization:** clip range [0.2, 0.3], KL loss disabled to encourage exploration.
73
 
74
  ---
75
 
@@ -93,48 +88,23 @@ wait()
93
  finished(content='xxx')
94
  ```
95
 
96
- ### Prompt template (action generator)
97
-
98
- ```
99
- You are a GUI agent. You are given a task and your action history, with
100
- screenshots. You need to perform the next action to complete the task.
101
-
102
- ## Output Format
103
- ```
104
- Thought: ...
105
- Action: ...
106
- ```
107
-
108
- ## Action Space
109
-
110
- {action_space}
111
-
112
- ## Note
113
- - Use {language} in `Thought` part.
114
- - Write a small plan and finally summarize your next action (with its target
115
- element) in one sentence in `Thought` part.
116
-
117
- ## User Instruction
118
- {instruction}
119
- ```
120
-
121
  ### Minimal inference example
122
 
123
  ```python
124
  from transformers import AutoProcessor, AutoModelForImageTextToText
125
  import torch
126
 
127
- model_id = "<your-username>/AgentHijack-Agent"
128
  processor = AutoProcessor.from_pretrained(model_id)
129
  model = AutoModelForImageTextToText.from_pretrained(
130
  model_id, torch_dtype=torch.bfloat16, device_map="auto"
131
  )
132
 
133
- # Build a chat with screenshot(s) + the action-generator prompt above,
134
  # then run model.generate(...) as usual.
135
  ```
136
 
137
- For the full agent framework (action generator + onlooker + environment checking), please refer to the code at [AgentHijack.github.io](https://AgentHijack.github.io).
138
 
139
  ---
140
 
@@ -156,4 +126,4 @@ If you use this model or the AgentHijack benchmark, please cite:
156
 
157
  ## Acknowledgements
158
 
159
- This model is built on top of [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) and the [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) family, with training infrastructure based on [VERL](https://github.com/volcengine/verl). The benchmark environment extends [OSWorld](https://os-world.github.io/).
 
1
  ---
 
 
 
2
  base_model:
3
  - ByteDance-Seed/UI-TARS-1.5-7B
4
+ language:
5
+ - en
6
+ library_name: transformers
7
+ license: apache-2.0
8
  pipeline_tag: image-text-to-text
9
  tags:
10
  - gui-agent
 
16
  - robustness
17
  - reinforcement-learning
18
  - grpo
 
19
  ---
20
 
21
  # AgentHijack-Agent
22
 
23
+ **AgentHijack-Agent** is the action-generation model released with the paper
24
+ [*AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions*](https://huggingface.co/papers/2605.25707) (ICML 2026).
25
 
26
  It is fine-tuned from [`UI-TARS-1.5-7B`](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) (Qwen2.5-VL architecture) using **Data-Augmented Group Relative Policy Optimization (DA-GRPO)** on the AgentHijack benchmark, with the goal of producing a computer-use agent that remains reliable under *common environment corruptions* (pop-ups, resolution changes, UI marks, subtitles, multi-apps, accidental touches, app minimization, network errors, and verification prompts).
27
 
28
  The same checkpoint serves a dual role in the AgentHijack-Agent framework:
 
29
  1. **Action generator** β€” produces the next GUI action from screenshots + history.
30
  2. **Onlooker** β€” summarizes behavioral changes between consecutive screenshots and performs an initial environment check before execution.
31
 
32
+ - πŸ“„ **Paper:** [AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions](https://huggingface.co/papers/2605.25707)
33
+ - 🌐 **Project page:** [https://AgentHijack.github.io](https://AgentHijack.github.io)
34
+ - πŸ’» **Code:** [https://github.com/tmlr-group/AgentHijack](https://github.com/tmlr-group/AgentHijack)
35
+ - πŸ‘₯ **Authors:** Jingwei Sun, Jianing Zhu, Yuanyi Li, Tongliang Liu, Xia Hu, and Bo Han
36
  - πŸ›οΈ **Affiliations:** TMLR Group, Hong Kong Baptist University
37
 
38
  ---
 
46
  - Recovers from **unexpected operations** (accidental touch, app minimization) via behavioral summarization.
47
  - Detects **environment errors** (network failure, login/verification prompts) up-front instead of looping on meaningless attempts.
48
 
 
 
49
  ---
50
 
51
  ## Model details
 
57
  | Precision | `bfloat16` |
58
  | Context length | 128k tokens |
59
  | Image resolution | 1920 Γ— 1080 (native, paper default) |
 
60
  | Tokenizer | Inherited from UI-TARS-1.5-7B / Qwen2.5-VL |
61
 
62
  ### Training
63
 
64
+ - **Algorithm:** Data-Augmented GRPO (DA-GRPO), an extension of GRPO that rolls out the same instruction across *different corrupted environments* drawn from a corruption set `C`.
65
  - **Framework:** [VERL](https://github.com/volcengine/verl).
66
+ - **Data:** 128 tasks sampled from the AgentHijack benchmark (built on top of OSWorld).
67
+ - **Reward:** `r = r_success + r_format`, with an experience-replay buffer to mitigate sparse-reward batches.
 
 
68
 
69
  ---
70
 
 
88
  finished(content='xxx')
89
  ```
90
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
91
  ### Minimal inference example
92
 
93
  ```python
94
  from transformers import AutoProcessor, AutoModelForImageTextToText
95
  import torch
96
 
97
+ model_id = "TMLR-Group-HF/AgentHijack-Agent"
98
  processor = AutoProcessor.from_pretrained(model_id)
99
  model = AutoModelForImageTextToText.from_pretrained(
100
  model_id, torch_dtype=torch.bfloat16, device_map="auto"
101
  )
102
 
103
+ # Build a chat with screenshot(s) + the action-generator prompt,
104
  # then run model.generate(...) as usual.
105
  ```
106
 
107
+ For the full agent framework (action generator + onlooker + environment checking), please refer to the code at [GitHub](https://github.com/tmlr-group/AgentHijack).
108
 
109
  ---
110
 
 
126
 
127
  ## Acknowledgements
128
 
129
+ This model is built on top of [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) and the [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) family, with training infrastructure based on [VERL](https://github.com/volcengine/verl). The benchmark environment extends [OSWorld](https://os-world.github.io/).