README.md

---
license: apache-2.0
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- vision-language
- mobile-gui-agent
- gui-safety
- world-model
- safety-guardrail
- action-risk-assessment
- semantic-next-state-prediction
- qwen3-vl
datasets:
- MobileWorld
- MobileSafetyBench
- MobileRisk
metrics:
- safety-utility-score
- risk-cost-score
- accuracy
- f1
---

# SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

**A plug-and-play, consequence-aware safety framework for securing mobile GUI agents before execution.**

[Project Website](https://seerguard.github.io/) | [Evaluation Code and Benchmarks](https://github.com/Autonomous-Agent-Team/SeerGuard.git)

SeerGuard protects mobile GUI agents before risky operations reach a real device. It introduces two pre-execution safeguards:

1. **Instruction-level screening** blocks explicitly malicious or unauthorized requests before task execution begins.
2. **Action-level risk assessment** predicts the consequence of each candidate action and blocks unsafe operations before they are executed.

At the core of SeerGuard is **SAWM**, a Safety-Augmented World Model built on Qwen3-VL-8B-Instruct. Given a mobile GUI screenshot, the user instruction, and a candidate action, SAWM predicts the semantic next state, evaluates the resulting risk, and returns a concise rationale.

This repository provides the **SAWM model weights**.

## Model Positioning

SeerGuard is designed as an **external guard model**, not as a replacement for the mobile GUI agent.

- **Plug-and-play integration:** SeerGuard can be inserted between an agent's planner and action executor as an independent safety checkpoint.
- **No fine-tuning of the protected agent:** Existing GUI-agent weights, planning policies, and action-generation logic do not need to be retrained or modified.
- **Agent-agnostic protection:** The same guardrail can be paired with different open- or closed-source GUI-agent backbones.
- **Pre-execution intervention:** Unsafe instructions and candidate actions are intercepted before they change the device state.
- **Low-friction deployment:** Integration only requires forwarding the instruction, current screenshot, and candidate action to the guard model before execution.

> **Clarification:** SAWM itself is fine-tuned from Qwen3-VL-8B-Instruct. "No fine-tuning" means that deploying SeerGuard does not require additional training or weight updates for the GUI agent being protected.

<a href="https://chat.qwenlm.ai/" target="_blank" style="margin: 2px;">
    <img alt="Chat" src="assets/overview.png" style="display: inline-block; vertical-align: middle;"/>
</a>

## Why SeerGuard

Mobile GUI agents interact with live applications, accounts, data, and device settings. A single incorrect tap may trigger an unauthorized payment, expose private information, send harmful content, delete data, or change a security-sensitive configuration.

Conventional post-hoc safety checkers review an interaction after the action has already occurred. SeerGuard instead performs **proactive consequence auditing**:

```text
GUI agent proposes an action
             |
             v
SeerGuard predicts the semantic consequence
             |
             v
Safety decision: safe or unsafe
```

This design enables the host agent to retain its original task-completion capabilities while SeerGuard independently handles safety assessment.

## Core Capabilities

### 1. Instruction-Level Screening

Before the GUI agent interacts with the device, SAWM determines whether the user request is explicitly malicious, unauthorized, or in conflict with core safety policies.

```text
Input:
- User instruction

Output:
- safety-judge: safe or unsafe,
- safety-reason: Brief explanation
```

Unsafe instructions are rejected before planning and execution.

### 2. Semantic Next-State Prediction

For each candidate action, SAWM predicts the likely functional consequence in natural language. It focuses on the semantic state transition rather than generating the next screen at the pixel level.

Examples include:

- a payment confirmation will be submitted;
- a message will be sent to a specific recipient;
- full-photo-library access will be granted;
- user data will be deleted;
- a security-sensitive setting will be changed.

### 3. Action-Level Risk Assessment

SAWM evaluates whether the predicted consequence is safe and provides a rationale for the decision.

```text
Input:
- Current mobile GUI screenshot
- User instruction
- Candidate action

Output:
- Current screen analysis
- Predicted semantic next state
- Safety rationale
- Action safety label: safe | unsafe
```

If the action is unsafe, SeerGuard blocks it before execution. Otherwise, the host agent proceeds normally.

The GUI agent remains responsible for perception, planning, and task completion. SeerGuard only decides whether a request or candidate action should be allowed to proceed.

## Model Architecture

SAWM uses **Qwen3-VL-8B-Instruct** as its backbone and is trained under a unified autoregressive formulation. It jointly learns:

- visual GUI understanding;
- malicious-instruction detection;
- semantic state-transition prediction;
- action-risk classification;
- natural-language safety rationale generation.

For action-level assessment, the model returns a structured response:

```json
{
  "ui_context_analysis": "...",
  "action_consequence_prediction": "...",
  "risk_evaluation": "...",
  "safety_judgment": "SAFE" | "UNSAFE"
}
```

## Training Data

SAWM is trained with a multi-task corpus that combines mobile world-modeling data with safety-augmentation data:

1. **General Textual Safety Data**
   - Provides broad safety alignment and malicious-instruction detection.

2. **Multimodal Mobile Risk Data**
   - Contains GUI trajectories with safety labels, semantic next-state descriptions, and rationales
   
3. **MobileWorld Next-State QA Data**
   - Provides mobile GUI state-transition and action-consequence prediction capability.

4. **Synthetic Textual Mobile Risk Data**
   - Bridges general text-only safety concepts and visually grounded mobile GUI risks.

The final training corpus contains approximately **148K instances**. SAWM is fine-tuned for **1 epoch** with a learning rate of **1e-6**.

## Evaluation

Full evaluation code and benchmark instructions are available in the [SeerGuard GitHub repository](https://github.com/Autonomous-Agent-Team/SeerGuard.git).

### Framework-Level Evaluation

On **MobileSafetyBench**, SeerGuard improves the safety-utility trade-off across Qwen3-VL, GPT-5.1, and Gemini-3.1 GUI-agent backbones.

For Qwen3-VL at a safety-priority setting of 0.8:

- Risk-Cost Score decreases from **0.347** to **0.130**.
- Safety-Utility Score increases from **0.191** to **0.596**.

[![MobileSafetyBench evaluation](assets/ms.png)](https://github.com/Autonomous-Agent-Team/SeerGuard.git)

### Instruction-Level Screening

SAWM is evaluated on Agent-SafetyBench and Prompt Injection benchmarks. On the Prompt Injection benchmark, SAWM achieves an F1 score of **0.922**, demonstrating strong risk detection while limiting unnecessary refusal of safe requests.

[![Instruction-level evaluation](assets/instruction.png)](https://github.com/Autonomous-Agent-Team/SeerGuard.git)

### Action-Level Risk Assessment

On **MobileRisk**, SAWM achieves:

- **F1: 0.723**
- **Step Score: 0.361**

Both are the best results among the compared methods, showing that SAWM can identify unsafe trajectories and locate the onset of risk.

[![MobileRisk evaluation](assets/mobilerisk.png)](https://github.com/Autonomous-Agent-Team/SeerGuard.git)

### Semantic Next-State Prediction

On Next-State-QA, SAWM reaches an accuracy of **0.762**, providing the predictive grounding required for consequence-aware action auditing.

## Intended Use

SAWM is intended for research and development of safety guardrails for mobile GUI agents.

Typical use cases include:

- proactive safety monitoring for mobile GUI agents;
- malicious-instruction filtering;
- pre-execution action-risk assessment;
- semantic next-state prediction;
- safety evaluation across different GUI-agent backbones;
- safety-utility trade-off analysis for autonomous mobile agents.


## Resources

- **Project Website:** https://seerguard.github.io/
- **Evaluation Code and Benchmarks:** https://github.com/Autonomous-Agent-Team/SeerGuard.git
- **Backbone Model:** Qwen3-VL-8B-Instruct

## Citation

If you find this model useful, please cite the SeerGuard paper:

```bibtex
@misc{yu2026seerguardsafetyframeworkmobile,
      title={SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction}, 
      author={Xue Yu and Bo Yuan and Pengshuai Yang and Kailin Zhao and Hong Hu and Junlan Feng},
      year={2026},
      eprint={2607.15550},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.15550}, 
}
```
```bibtex
@inproceedings{seerguard2026,
  title     = {SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction},
  author    = {Xue Yu and Bo Yuan and Pengshuai Yang and Kailin Zhao and Hong Hu and Junlan Feng},
  booktitle = {Proceedings of the ACM Multimedia Conference},
  year      = {2026},
  note      = {To appear}
}
```

SAWM is built on Qwen3-VL-8B-Instruct. Please also cite the corresponding Qwen-VL technical report when using this model.

```bibtex
@misc{qwen3technicalreport,
  title         = {Qwen3 Technical Report},
  author        = {Qwen Team},
  year          = {2025},
  eprint        = {2505.09388},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2505.09388}
}
```