README.md

---
license: cc-by-nc-nd-4.0
tags:
- ai-safety
- agent-safety
- agent-security
- mobile-agent
---
# MATE-3B

<div align="center">

<img src=https://raw.githubusercontent.com/jiangchangyue/MATE/main/assets/figures/logo.png width="28%"/> 

[![Usenix Security 2026 Accepted](https://img.shields.io/badge/UsenixSecurity-2026-green.svg)]()
[![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-blue.svg)](LICENSE)

[中文 README](README_CN.md)

</div>

# MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning

Changyue Jiang, Jiayi Wang, Xin Wen, Jiarun Dai, Geng Hong, and Xudong Pan

Shanghai Innovation Institute and Fudan University

Paper (coming soon) | Dataset (coming soon) 

**GitHub**：https://github.com/jiangchangyue/MATE

## Overview

Foundation-model-powered mobile agents can execute complex workflows on real devices, but their trajectories may violate app-specific, organization-specific, or user-defined security policies. Existing defenses typically use brittle static rules or prompt a general-purpose LLM as a judge.

**MATE** is a lightweight, policy-aware trajectory auditor. Given an instruction, an agent trajectory, and natural-language security policies, MATE outputs:

- a policy-conditioned violation decision;
- one of 14 mobile-agent risk categories; and
- a natural-language rationale grounded in the relevant trajectory steps.

Policies remain editable text rather than fixed model parameters, so security requirements can be updated without retraining the auditor.


<div align=center>
<img src="https://raw.githubusercontent.com/jiangchangyue/MATE/main/assets/figures/mate_methodology.png" width="88%"/>
</div>

The MATE pipeline has four stages:

1. **App information collection:** collect functional descriptions, operation workflows, and security policies from 158 Chinese and English mobile apps.
2. **Knowledge-grounded synthesis:** generate instructions, ReAct-style trajectories, policy-conditioned labels, risk categories, and explanations.
3. **Data augmentation and training:** introduce trajectory-policy mismatch, multi-policy supervision, and multi-app workflows, then fine-tune 0.5B, 1.5B, and 3B auditors.
4. **Deployment:** normalize heterogeneous raw logs with a trajectory adapter and automatically retrieve relevant policies for auditing.

The final training corpus contains more than **140K** semantically realistic, policy-conditioned trajectories.

## MATEBench

We introduce **MATEBench**, a bilingual Chinese-English benchmark for policy-aware, trajectory-level mobile-agent security auditing. It covers all 14 risk categories and contains three subsets:

| Subset | Distribution | Apps | Trajectories | Evaluation goal |
| --- | --- | ---: | ---: | --- |
| **MATEBench-In** | In-domain, synthetic | 134 seen apps | 2,775 | Controlled evaluation on apps represented during training |
| **MATEBench-Out** | Out-of-domain, synthetic | 24 unseen apps | 1,150 | Generalization to new app semantics and workflows |
| **MATEBench-Real** | Out-of-domain, real device | 13 apps / 3 agents | 162 | Auditing real trajectories with realistic logging noise |

The real subset includes trajectories from Zhipu's AutoGLM, Alibaba's Mobile-Agent, and an Android-emulator-based agent. Synthetic and real subsets have closely aligned trajectory lengths and GUI-action distributions.

## Main Results

MATE consistently outperforms a static rule engine, general-purpose LLM judges, ShieldAgent, and ShieldLM on MATEBench, R-Judge, and ASSEBench.

<!-- ![Overall accuracy and F1 results](assets/figures/eval-result.png) -->
<div align=center>
<img src="https://raw.githubusercontent.com/jiangchangyue/MATE/main/assets/figures/eval-result.png" width="88%"/>
</div>

MATE-3B achieves:

| Benchmark | Accuracy | F1 |
| --- | ---: | ---: |
| R-Judge | 92.64% | 93.92% |
| ASSEBench | 94.98% | 95.54% |
| MATEBench-In | 96.83% | 95.53% |
| MATEBench-Out | 95.48% | 94.68% |
| MATEBench-Real | 95.06% | 95.60% |

Even MATE-0.5B surpasses strong prompt-based evaluators on MATEBench-Out while requiring only **0.09 seconds per trajectory** on a single NVIDIA H100.

<div align=center>
<img src="https://raw.githubusercontent.com/jiangchangyue/MATE/main/assets/figures/ablation-table-1.png" alt="Fine-tuning ablation across MATEBench" width="60%">
</div>

## Ablation Studies

Zero-shot Qwen2.5-Instruct models achieve less than 50% accuracy on average for policy-conditioned trajectory auditing. Fine-tuning on the synthesized MATE corpus raises overall accuracy above 92%, demonstrating the importance of task-specific trajectory learning.

<!-- ![Fine-tuning ablation across MATEBench](assets/figures/ablation_v4_filled.png) -->
<div align=center>
<img src="https://raw.githubusercontent.com/jiangchangyue/MATE/main/assets/figures/ablation_v4_filled.png" width="88%"/>
</div>

External app knowledge and multi-stage quality repair are both necessary. In the controlled MATE-0.5B ablation on MATEBench-Real, the full synthesis pipeline reaches 75.93% accuracy and 79.14% F1, while removing both components reduces accuracy to 46.91%.

## Real-World Auditing

MATE-3B successfully audits trajectories from deployed mobile agents and produces explicit policy-grounded explanations. The following cases identify a mass-forwarded malicious coupon link as **Becoming a Fraud Relay** and access to a malware-test page as **Device Security Compromised**.

<!-- ![Real-world mobile-agent trajectories audited by MATE-3B](assets/figures/mate-case.png) -->
<div align=center>
<img src="https://raw.githubusercontent.com/jiangchangyue/MATE/main/assets/figures/mate-case.png" width="88%"/>
</div>

---

## Installation and Usage 

# GitHub: https://github.com/jiangchangyue/MATE


This repository contains the MATE implementation for benchmark evaluation, data synthesis and augmentation,
trajectory normalization, and policy retrieval. The large benchmark files and model checkpoints must be
downloaded separately and placed in the paths described below.

<!-- ### Repository Layout

```text
.
├── MateBench/                         # MATEBench-In, -Out, and -Real
├── Mate_model/                        # MATE-0.5B, MATE-1.5B, and MATE-3B checkpoints
├── data_synthesis_pipeline/
│   ├── appinfo/                       # App workflows and security policies
│   ├── traj_synthesis/                # Instruction and trajectory synthesis
│   └── data_augmentation/
│       ├── multi-app/
│       ├── multi-policy/
│       └── traj_policy_mismatched/
├── evaluation/
│   ├── run_matebench.py               # Unified benchmark entry point
│   ├── matebench_eval.py              # Metrics and evaluation logic
│   └── launch_mate_vllm.sh            # Local vLLM server launcher
├── model_deployment/
│   └── get_traj_for_mate.py           # Adapter and retriever pipeline
├── policy_retriever/
│   ├── policy_retriever.py
│   ├── model/policy_retriever/
│   └── policy_pool/
├── trajectory_adapter/
│   └── trajectory_adapter.py
├── run_minimal_demo.sh
├── run_eval_subset.sh
├── summarize_results.py
└── requirements.txt
``` -->

### 1. Environment Setup

The implementation was developed with Python 3.11. A CUDA-capable Linux environment is recommended when
serving MATE checkpoints locally with vLLM.

```bash
conda create -n mate python=3.11 -y
conda activate mate

python -m pip install --upgrade pip
pip install -r requirements.txt
```

The main dependencies include:

- `vllm` and `openai` for OpenAI-compatible model serving and requests;
- `torch` and `sentence-transformers` for policy retrieval;
- `langchain-huggingface` or `langchain-community` for embedding utilities;
- `json_repair` for synthesis and augmentation output parsing; and
- `scikit-learn` and `matplotlib` for rationale-similarity visualization.

If MATE is accessed through an existing remote OpenAI-compatible endpoint, a local vLLM server is not
required. In that case, install the remaining dependencies in an environment supported by your platform.

### 2. Prepare Models and Data

Before running the complete examples, place the downloaded artifacts at the following locations:

```text
Mate_model/
├── MATE-0.5B/
├── MATE-1.5B/
└── MATE-3B/

MateBench/
├── MateBench-In/
├── MateBench-Out/
└── MateBench-Real/

policy_retriever/model/policy_retriever/
```

Each MATE checkpoint should follow the Hugging Face Transformers layout and contain its model configuration,
tokenizer, generation configuration, and `safetensors` weights. The policy-retriever directory must contain a
local SentenceTransformers-compatible model.

The evaluator uses `BAAI/bge-large-zh-v1.5` to calculate rationale similarity. In an offline environment,
pre-cache this model or change the embedding path in `evaluation/matebench_eval.py` to a local copy.

You can quickly verify whether the required directories are populated:

```bash
find Mate_model -maxdepth 2 -type f | head
find MateBench -type f -name '*.json' | head
find policy_retriever/model/policy_retriever -maxdepth 2 -type f | head
```

### 3. MATEBench Data Format

Each benchmark example is a JSON object containing an instruction, a ReAct-style trajectory, one or more
security policies, and a policy-conditioned reference result:

```json
{
  "instruction": "A user instruction",
  "trajectory": [
    {
      "Thought": "Agent reasoning",
      "Action": "click",
      "Action Input": {"x": 100, "y": 200},
      "Observation": "The next UI state"
    },
    {
      "Thought": "The task is complete",
      "Final Answer": "Completion message"
    }
  ],
  "safe_rule": "A natural-language security policy",
  "evaluation_result": {
    "Violation": "yes",
    "Category": "Privacy Leakage",
    "Rationale": "Why the trajectory violates the policy"
  }
}
```

### 4. Quick Start

#### Validate subset construction without a model

This command checks the benchmark paths and creates a two-example temporary subset without starting a model
server:

```bash
CREATE_ONLY=1 \
SUBSET_SIZE=2 \
MATE_TASK=in \
MATE_LANG=en \
bash run_eval_subset.sh
```

The temporary subset is written under `/tmp/matebench_subset_*`.

#### Run the minimal end-to-end demo

The smallest complete demo starts MATE-1.5B with vLLM, creates a small English MATEBench-In subset, evaluates
it, and prints a Markdown metrics summary:

```bash
bash run_minimal_demo.sh
```

Default configuration:

```text
MODEL_PATH=Mate_model/MATE-1.5B
SERVED_MODEL=MATE-1.5B
PORT=8016
MATE_TASK=in
MATE_LANG=en
SUBSET_SIZE=50
BATCH_SIZE=8
```

Override settings with environment variables:

```bash
MODEL_PATH=Mate_model/MATE-0.5B \
SERVED_MODEL=MATE-0.5B \
SUBSET_SIZE=10 \
BATCH_SIZE=4 \
PORT=8016 \
bash run_minimal_demo.sh
```

The server started by the script is stopped when evaluation finishes. Set `KEEP_SERVER=1` to keep it running.

#### Use an existing OpenAI-compatible endpoint

If a compatible server is already available at `http://127.0.0.1:8016/v1`, skip server startup:

```bash
START_SERVER=0 \
MODEL_NAME=MATE-1.5B \
BASE_URL=http://127.0.0.1:8016/v1 \
API_KEY=EMPTY \
MATE_TASK=in \
MATE_LANG=en \
SUBSET_SIZE=50 \
BATCH_SIZE=8 \
bash run_eval_subset.sh
```

### 5. Serve a MATE Checkpoint

Start MATE-3B on port 8016:

```bash
bash evaluation/launch_mate_vllm.sh \
  Mate_model/MATE-3B \
  MATE-3B \
  8016
```

The launcher is equivalent to:

```bash
python -m vllm.entrypoints.openai.api_server \
  --model Mate_model/MATE-3B \
  --served-model-name MATE-3B \
  --max-model-len 8192 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.8 \
  --port 8016
```

For multiple GPUs or a smaller context window:

```bash
TP=2 \
GPU_UTIL=0.9 \
MAX_LEN=8192 \
bash evaluation/launch_mate_vllm.sh Mate_model/MATE-3B MATE-3B 8016
```

Confirm that the endpoint is ready:

```bash
curl http://127.0.0.1:8016/v1/models
```

### 6. Run Full MATEBench Evaluation

Configure the evaluator through environment variables:

```bash
export MATE_MODEL_NAME=MATE-3B
export MATE_BASE_URL=http://127.0.0.1:8016/v1
export MATE_API_KEY=EMPTY
```

Evaluate the in-domain, out-of-domain, and real-world subsets:

```bash
python evaluation/run_matebench.py \
  --task in \
  --lang all \
  --batch-size 64 \
  --model MATE-3B

python evaluation/run_matebench.py \
  --task out \
  --lang all \
  --batch-size 64 \
  --model MATE-3B

python evaluation/run_matebench.py \
  --task real \
  --lang all \
  --batch-size 64 \
  --model MATE-3B
```

Supported arguments:

| Argument | Values | Description |
| --- | --- | --- |
| `--task` | `in`, `out`, `real` | MATEBench subset |
| `--lang` | `all`, `cn`, `en` | Language split |
| `--batch-size` | positive integer | Number of concurrent requests |
| `--model` | served model name | Overrides `MATE_MODEL_NAME` |
| `--data-root` | directory | Benchmark root; default: `./MateBench` |
| `--output-root` | directory | Result root; default: `./evaluation/evaluation_results` |
| `--no-visualize` | flag | Skip PCA rationale visualizations |

Example with custom paths:

```bash
python evaluation/run_matebench.py \
  --task in \
  --lang en \
  --batch-size 16 \
  --model MATE-3B \
  --data-root ./MateBench \
  --output-root ./evaluation/evaluation_results \
  --no-visualize
```

Results are stored as:

```text
evaluation/evaluation_results/
└── matebench_{in,out,real}_results/
    └── <model-name>/
        └── <YYYYMMDD_HHMMSS>_<lang>/
            ├── eval_results.txt
            ├── eval_model_output.json
            ├── rationale_semantic.pdf
            └── rationale_semantic.png
```

Summarize all recorded runs or only the latest run:

```bash
python summarize_results.py --root evaluation/evaluation_results
python summarize_results.py --root evaluation/evaluation_results --latest
```

### 7. Data Synthesis

The synthesis modules call an OpenAI-compatible endpoint. Configure the `llm_config.py` inside the module
that you are running:

```python
MODELS_CONFIG = {
    "llm": {
        "model_name": "your-model-name",
        "base_url": "http://your-api-endpoint/v1",
        "api_key": "your-api-key",
    }
}
```

#### Generate task instructions

```bash
cd data_synthesis_pipeline/traj_synthesis

python main.py \
  --task instruction \
  --type en \
  --num 200 \
  --batch_size 50 \
  --pairs_per_call 20 \
  --output_path ./synthesis-results
```

#### Generate trajectories

```bash
cd data_synthesis_pipeline/traj_synthesis

python main.py \
  --task trajectory \
  --type en \
  --num 110 \
  --traj_batch_size 100 \
  --output_path ./synthesis-results
```

Use `--type cn` for Chinese generation. The provided `instructions_synthesis.sh` and
`trajectory_synthesis.sh` scripts contain compact versions of these commands.

### 8. Data Augmentation

All augmentation modules are under `data_synthesis_pipeline/data_augmentation/`.

#### Multi-app trajectories

```bash
cd data_synthesis_pipeline/data_augmentation/multi-app

python main_multiapp.py \
  --task instruction \
  --type en \
  --num 110 \
  --batch_size 50 \
  --output_path ./synthesis-multiapp

python main_multiapp.py \
  --task trajectory \
  --type en \
  --num 20 \
  --traj_batch_size 10 \
  --output_path ./synthesis-multiapp
```

#### Multi-policy supervision

```bash
cd data_synthesis_pipeline/data_augmentation/multi-policy

python multipolicy_data.py \
  --input_path ../../traj_synthesis/synthesis-results/results-en/trajectory \
  --num 100 \
  --batch_size 100 \
  --similarity_threshold 0.5 \
  --include_original yes \
  --output_path ./multipolicy_output-en-yes \
  --embedding_model_path ../../../policy_retriever/model/policy_retriever
```

Set `--include_original yes` to retain the matched policy and append additional policies. Set it to `no` to
construct a policy set that excludes the original matched policy.

#### Trajectory-policy mismatch

```bash
cd data_synthesis_pipeline/data_augmentation/traj_policy_mismatched

python main_batch.py \
  --input_path ../../traj_synthesis/synthesis-results/results-en/trajectory \
  --num 100 \
  --batch_size 100 \
  --similarity_threshold 0.5 \
  --output_path ./output-en \
  --embedding_model_path ../../../policy_retriever/model/policy_retriever
```

The mismatch pipeline pairs trajectories with semantically unrelated policies to create hard negative
examples and reduce false-positive auditing decisions.

### 9. Trajectory Adapter and Policy Retriever

`model_deployment/get_traj_for_mate.py` converts raw agent logs into MATE's canonical trajectory schema and
retrieves the most relevant security policy.

For text trajectories:

```bash
mkdir -p model_deployment/input_test/text model_deployment/output_test/text
# Place one JSON object per file under model_deployment/input_test/text.

python model_deployment/get_traj_for_mate.py \
  --input_path model_deployment/input_test/text \
  --lang en \
  --input_type text \
  --output_dir model_deployment/output_test/text \
  --adapter_batch_size 10 \
  --adapter_base_url http://127.0.0.1:8016/v1 \
  --adapter_model_name your-model-name \
  --adapter_api_key EMPTY \
  --retriever_model_path policy_retriever/model/policy_retriever \
  --retriever_batch_size 128 \
  --rule_pool_en policy_retriever/policy_pool/policy_en.jsonl \
  --rule_pool_cn policy_retriever/policy_pool/policy_cn.jsonl
```

For screenshot-based or multimodal trajectories:

```bash
mkdir -p model_deployment/input_test/mm model_deployment/output_test/mm
# Place the multimodal JSON inputs and referenced images under the input directory.

python model_deployment/get_traj_for_mate.py \
  --input_path model_deployment/input_test/mm \
  --lang en \
  --input_type multimodal \
  --output_dir model_deployment/output_test/mm \
  --mm_base_url http://127.0.0.1:8016/v1 \
  --mm_model_name your-vlm-name \
  --mm_api_key EMPTY \
  --retriever_model_path policy_retriever/model/policy_retriever \
  --retriever_batch_size 128 \
  --rule_pool_en policy_retriever/policy_pool/policy_en.jsonl \
  --rule_pool_cn policy_retriever/policy_pool/policy_cn.jsonl \
  --log_model_io \
  --log_dir model_deployment/logs
```

The input path must be a directory containing JSON files. Inputs already matching the MATE schema skip
trajectory conversion but still pass through policy retrieval.

### 10. Troubleshooting

**The benchmark smoke test reports missing directories**

Populate `MateBench/MateBench-In`, `MateBench/MateBench-Out`, and `MateBench/MateBench-Real` with the released
JSON files before running `run_eval_subset.sh`.

**vLLM runs out of GPU memory**

Start with MATE-0.5B and reduce context length or memory utilization:

```bash
GPU_UTIL=0.7 \
MAX_LEN=4096 \
bash evaluation/launch_mate_vllm.sh Mate_model/MATE-0.5B MATE-0.5B 8016
```

**Evaluation is too slow**

Reduce request concurrency:

```bash
python evaluation/run_matebench.py \
  --task in \
  --lang en \
  --batch-size 8 \
  --model MATE-1.5B
```

**The evaluator cannot connect to the model**

```bash
curl http://127.0.0.1:8016/v1/models
```

Use `127.0.0.1` for a client on the same machine. Although vLLM may bind to `0.0.0.0`, clients should not use
`0.0.0.0` as the destination address.

**The retriever cannot find its model**

Confirm that `policy_retriever/model/policy_retriever/` contains a valid SentenceTransformers model and that
both policy pools exist:

```bash
ls policy_retriever/model/policy_retriever
ls policy_retriever/policy_pool/policy_{cn,en}.jsonl
```

**Offline rationale evaluation fails**

Pre-download `BAAI/bge-large-zh-v1.5` and configure `evaluation/matebench_eval.py` to use its local path.

<!-- ## Repository Structure

```text
.
├── index.html              # Project homepage
├── paper.pdf               # Paper manuscript
└── assets
    ├── css/style.css       # Homepage styles
    └── figures             # Method, results, ablations, and case studies
``` -->

## Citation

```bibtex
@misc{jiang2026mate,
  title={MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning},
  author={Jiang, Changyue and Wang, Jiayi and Wen, Xin and Dai, Jiarun and Hong, Geng and Pan, Xudong},
  year={2026}
}
```