--- license: cc-by-nc-nd-4.0 tags: - ai-safety - agent-safety - agent-security - mobile-agent --- # MATE-3B
[![Usenix Security 2026 Accepted](https://img.shields.io/badge/UsenixSecurity-2026-green.svg)]() [![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-blue.svg)](LICENSE) [中文 README](README_CN.md)
# MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning Changyue Jiang, Jiayi Wang, Xin Wen, Jiarun Dai, Geng Hong, and Xudong Pan Shanghai Innovation Institute and Fudan University Paper (coming soon) | Dataset (coming soon) **GitHub**：https://github.com/jiangchangyue/MATE ## Overview Foundation-model-powered mobile agents can execute complex workflows on real devices, but their trajectories may violate app-specific, organization-specific, or user-defined security policies. Existing defenses typically use brittle static rules or prompt a general-purpose LLM as a judge. **MATE** is a lightweight, policy-aware trajectory auditor. Given an instruction, an agent trajectory, and natural-language security policies, MATE outputs: - a policy-conditioned violation decision; - one of 14 mobile-agent risk categories; and - a natural-language rationale grounded in the relevant trajectory steps. Policies remain editable text rather than fixed model parameters, so security requirements can be updated without retraining the auditor.
The MATE pipeline has four stages: 1. **App information collection:** collect functional descriptions, operation workflows, and security policies from 158 Chinese and English mobile apps. 2. **Knowledge-grounded synthesis:** generate instructions, ReAct-style trajectories, policy-conditioned labels, risk categories, and explanations. 3. **Data augmentation and training:** introduce trajectory-policy mismatch, multi-policy supervision, and multi-app workflows, then fine-tune 0.5B, 1.5B, and 3B auditors. 4. **Deployment:** normalize heterogeneous raw logs with a trajectory adapter and automatically retrieve relevant policies for auditing. The final training corpus contains more than **140K** semantically realistic, policy-conditioned trajectories. ## MATEBench We introduce **MATEBench**, a bilingual Chinese-English benchmark for policy-aware, trajectory-level mobile-agent security auditing. It covers all 14 risk categories and contains three subsets: | Subset | Distribution | Apps | Trajectories | Evaluation goal | | --- | --- | ---: | ---: | --- | | **MATEBench-In** | In-domain, synthetic | 134 seen apps | 2,775 | Controlled evaluation on apps represented during training | | **MATEBench-Out** | Out-of-domain, synthetic | 24 unseen apps | 1,150 | Generalization to new app semantics and workflows | | **MATEBench-Real** | Out-of-domain, real device | 13 apps / 3 agents | 162 | Auditing real trajectories with realistic logging noise | The real subset includes trajectories from Zhipu's AutoGLM, Alibaba's Mobile-Agent, and an Android-emulator-based agent. Synthetic and real subsets have closely aligned trajectory lengths and GUI-action distributions. ## Main Results MATE consistently outperforms a static rule engine, general-purpose LLM judges, ShieldAgent, and ShieldLM on MATEBench, R-Judge, and ASSEBench.
MATE-3B achieves: | Benchmark | Accuracy | F1 | | --- | ---: | ---: | | R-Judge | 92.64% | 93.92% | | ASSEBench | 94.98% | 95.54% | | MATEBench-In | 96.83% | 95.53% | | MATEBench-Out | 95.48% | 94.68% | | MATEBench-Real | 95.06% | 95.60% | Even MATE-0.5B surpasses strong prompt-based evaluators on MATEBench-Out while requiring only **0.09 seconds per trajectory** on a single NVIDIA H100.
Fine-tuning ablation across MATEBench
## Ablation Studies Zero-shot Qwen2.5-Instruct models achieve less than 50% accuracy on average for policy-conditioned trajectory auditing. Fine-tuning on the synthesized MATE corpus raises overall accuracy above 92%, demonstrating the importance of task-specific trajectory learning.
External app knowledge and multi-stage quality repair are both necessary. In the controlled MATE-0.5B ablation on MATEBench-Real, the full synthesis pipeline reaches 75.93% accuracy and 79.14% F1, while removing both components reduces accuracy to 46.91%. ## Real-World Auditing MATE-3B successfully audits trajectories from deployed mobile agents and produces explicit policy-grounded explanations. The following cases identify a mass-forwarded malicious coupon link as **Becoming a Fraud Relay** and access to a malware-test page as **Device Security Compromised**.
--- ## Installation and Usage # GitHub: https://github.com/jiangchangyue/MATE This repository contains the MATE implementation for benchmark evaluation, data synthesis and augmentation, trajectory normalization, and policy retrieval. The large benchmark files and model checkpoints must be downloaded separately and placed in the paths described below. ### 1. Environment Setup The implementation was developed with Python 3.11. A CUDA-capable Linux environment is recommended when serving MATE checkpoints locally with vLLM. ```bash conda create -n mate python=3.11 -y conda activate mate python -m pip install --upgrade pip pip install -r requirements.txt ``` The main dependencies include: - `vllm` and `openai` for OpenAI-compatible model serving and requests; - `torch` and `sentence-transformers` for policy retrieval; - `langchain-huggingface` or `langchain-community` for embedding utilities; - `json_repair` for synthesis and augmentation output parsing; and - `scikit-learn` and `matplotlib` for rationale-similarity visualization. If MATE is accessed through an existing remote OpenAI-compatible endpoint, a local vLLM server is not required. In that case, install the remaining dependencies in an environment supported by your platform. ### 2. Prepare Models and Data Before running the complete examples, place the downloaded artifacts at the following locations: ```text Mate_model/ ├── MATE-0.5B/ ├── MATE-1.5B/ └── MATE-3B/ MateBench/ ├── MateBench-In/ ├── MateBench-Out/ └── MateBench-Real/ policy_retriever/model/policy_retriever/ ``` Each MATE checkpoint should follow the Hugging Face Transformers layout and contain its model configuration, tokenizer, generation configuration, and `safetensors` weights. The policy-retriever directory must contain a local SentenceTransformers-compatible model. The evaluator uses `BAAI/bge-large-zh-v1.5` to calculate rationale similarity. In an offline environment, pre-cache this model or change the embedding path in `evaluation/matebench_eval.py` to a local copy. You can quickly verify whether the required directories are populated: ```bash find Mate_model -maxdepth 2 -type f | head find MateBench -type f -name '*.json' | head find policy_retriever/model/policy_retriever -maxdepth 2 -type f | head ``` ### 3. MATEBench Data Format Each benchmark example is a JSON object containing an instruction, a ReAct-style trajectory, one or more security policies, and a policy-conditioned reference result: ```json { "instruction": "A user instruction", "trajectory": [ { "Thought": "Agent reasoning", "Action": "click", "Action Input": {"x": 100, "y": 200}, "Observation": "The next UI state" }, { "Thought": "The task is complete", "Final Answer": "Completion message" } ], "safe_rule": "A natural-language security policy", "evaluation_result": { "Violation": "yes", "Category": "Privacy Leakage", "Rationale": "Why the trajectory violates the policy" } } ``` ### 4. Quick Start #### Validate subset construction without a model This command checks the benchmark paths and creates a two-example temporary subset without starting a model server: ```bash CREATE_ONLY=1 \ SUBSET_SIZE=2 \ MATE_TASK=in \ MATE_LANG=en \ bash run_eval_subset.sh ``` The temporary subset is written under `/tmp/matebench_subset_*`. #### Run the minimal end-to-end demo The smallest complete demo starts MATE-1.5B with vLLM, creates a small English MATEBench-In subset, evaluates it, and prints a Markdown metrics summary: ```bash bash run_minimal_demo.sh ``` Default configuration: ```text MODEL_PATH=Mate_model/MATE-1.5B SERVED_MODEL=MATE-1.5B PORT=8016 MATE_TASK=in MATE_LANG=en SUBSET_SIZE=50 BATCH_SIZE=8 ``` Override settings with environment variables: ```bash MODEL_PATH=Mate_model/MATE-0.5B \ SERVED_MODEL=MATE-0.5B \ SUBSET_SIZE=10 \ BATCH_SIZE=4 \ PORT=8016 \ bash run_minimal_demo.sh ``` The server started by the script is stopped when evaluation finishes. Set `KEEP_SERVER=1` to keep it running. #### Use an existing OpenAI-compatible endpoint If a compatible server is already available at `http://127.0.0.1:8016/v1`, skip server startup: ```bash START_SERVER=0 \ MODEL_NAME=MATE-1.5B \ BASE_URL=http://127.0.0.1:8016/v1 \ API_KEY=EMPTY \ MATE_TASK=in \ MATE_LANG=en \ SUBSET_SIZE=50 \ BATCH_SIZE=8 \ bash run_eval_subset.sh ``` ### 5. Serve a MATE Checkpoint Start MATE-3B on port 8016: ```bash bash evaluation/launch_mate_vllm.sh \ Mate_model/MATE-3B \ MATE-3B \ 8016 ``` The launcher is equivalent to: ```bash python -m vllm.entrypoints.openai.api_server \ --model Mate_model/MATE-3B \ --served-model-name MATE-3B \ --max-model-len 8192 \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.8 \ --port 8016 ``` For multiple GPUs or a smaller context window: ```bash TP=2 \ GPU_UTIL=0.9 \ MAX_LEN=8192 \ bash evaluation/launch_mate_vllm.sh Mate_model/MATE-3B MATE-3B 8016 ``` Confirm that the endpoint is ready: ```bash curl http://127.0.0.1:8016/v1/models ``` ### 6. Run Full MATEBench Evaluation Configure the evaluator through environment variables: ```bash export MATE_MODEL_NAME=MATE-3B export MATE_BASE_URL=http://127.0.0.1:8016/v1 export MATE_API_KEY=EMPTY ``` Evaluate the in-domain, out-of-domain, and real-world subsets: ```bash python evaluation/run_matebench.py \ --task in \ --lang all \ --batch-size 64 \ --model MATE-3B python evaluation/run_matebench.py \ --task out \ --lang all \ --batch-size 64 \ --model MATE-3B python evaluation/run_matebench.py \ --task real \ --lang all \ --batch-size 64 \ --model MATE-3B ``` Supported arguments: | Argument | Values | Description | | --- | --- | --- | | `--task` | `in`, `out`, `real` | MATEBench subset | | `--lang` | `all`, `cn`, `en` | Language split | | `--batch-size` | positive integer | Number of concurrent requests | | `--model` | served model name | Overrides `MATE_MODEL_NAME` | | `--data-root` | directory | Benchmark root; default: `./MateBench` | | `--output-root` | directory | Result root; default: `./evaluation/evaluation_results` | | `--no-visualize` | flag | Skip PCA rationale visualizations | Example with custom paths: ```bash python evaluation/run_matebench.py \ --task in \ --lang en \ --batch-size 16 \ --model MATE-3B \ --data-root ./MateBench \ --output-root ./evaluation/evaluation_results \ --no-visualize ``` Results are stored as: ```text evaluation/evaluation_results/ └── matebench_{in,out,real}_results/ └── / └── _/ ├── eval_results.txt ├── eval_model_output.json ├── rationale_semantic.pdf └── rationale_semantic.png ``` Summarize all recorded runs or only the latest run: ```bash python summarize_results.py --root evaluation/evaluation_results python summarize_results.py --root evaluation/evaluation_results --latest ``` ### 7. Data Synthesis The synthesis modules call an OpenAI-compatible endpoint. Configure the `llm_config.py` inside the module that you are running: ```python MODELS_CONFIG = { "llm": { "model_name": "your-model-name", "base_url": "http://your-api-endpoint/v1", "api_key": "your-api-key", } } ``` #### Generate task instructions ```bash cd data_synthesis_pipeline/traj_synthesis python main.py \ --task instruction \ --type en \ --num 200 \ --batch_size 50 \ --pairs_per_call 20 \ --output_path ./synthesis-results ``` #### Generate trajectories ```bash cd data_synthesis_pipeline/traj_synthesis python main.py \ --task trajectory \ --type en \ --num 110 \ --traj_batch_size 100 \ --output_path ./synthesis-results ``` Use `--type cn` for Chinese generation. The provided `instructions_synthesis.sh` and `trajectory_synthesis.sh` scripts contain compact versions of these commands. ### 8. Data Augmentation All augmentation modules are under `data_synthesis_pipeline/data_augmentation/`. #### Multi-app trajectories ```bash cd data_synthesis_pipeline/data_augmentation/multi-app python main_multiapp.py \ --task instruction \ --type en \ --num 110 \ --batch_size 50 \ --output_path ./synthesis-multiapp python main_multiapp.py \ --task trajectory \ --type en \ --num 20 \ --traj_batch_size 10 \ --output_path ./synthesis-multiapp ``` #### Multi-policy supervision ```bash cd data_synthesis_pipeline/data_augmentation/multi-policy python multipolicy_data.py \ --input_path ../../traj_synthesis/synthesis-results/results-en/trajectory \ --num 100 \ --batch_size 100 \ --similarity_threshold 0.5 \ --include_original yes \ --output_path ./multipolicy_output-en-yes \ --embedding_model_path ../../../policy_retriever/model/policy_retriever ``` Set `--include_original yes` to retain the matched policy and append additional policies. Set it to `no` to construct a policy set that excludes the original matched policy. #### Trajectory-policy mismatch ```bash cd data_synthesis_pipeline/data_augmentation/traj_policy_mismatched python main_batch.py \ --input_path ../../traj_synthesis/synthesis-results/results-en/trajectory \ --num 100 \ --batch_size 100 \ --similarity_threshold 0.5 \ --output_path ./output-en \ --embedding_model_path ../../../policy_retriever/model/policy_retriever ``` The mismatch pipeline pairs trajectories with semantically unrelated policies to create hard negative examples and reduce false-positive auditing decisions. ### 9. Trajectory Adapter and Policy Retriever `model_deployment/get_traj_for_mate.py` converts raw agent logs into MATE's canonical trajectory schema and retrieves the most relevant security policy. For text trajectories: ```bash mkdir -p model_deployment/input_test/text model_deployment/output_test/text # Place one JSON object per file under model_deployment/input_test/text. python model_deployment/get_traj_for_mate.py \ --input_path model_deployment/input_test/text \ --lang en \ --input_type text \ --output_dir model_deployment/output_test/text \ --adapter_batch_size 10 \ --adapter_base_url http://127.0.0.1:8016/v1 \ --adapter_model_name your-model-name \ --adapter_api_key EMPTY \ --retriever_model_path policy_retriever/model/policy_retriever \ --retriever_batch_size 128 \ --rule_pool_en policy_retriever/policy_pool/policy_en.jsonl \ --rule_pool_cn policy_retriever/policy_pool/policy_cn.jsonl ``` For screenshot-based or multimodal trajectories: ```bash mkdir -p model_deployment/input_test/mm model_deployment/output_test/mm # Place the multimodal JSON inputs and referenced images under the input directory. python model_deployment/get_traj_for_mate.py \ --input_path model_deployment/input_test/mm \ --lang en \ --input_type multimodal \ --output_dir model_deployment/output_test/mm \ --mm_base_url http://127.0.0.1:8016/v1 \ --mm_model_name your-vlm-name \ --mm_api_key EMPTY \ --retriever_model_path policy_retriever/model/policy_retriever \ --retriever_batch_size 128 \ --rule_pool_en policy_retriever/policy_pool/policy_en.jsonl \ --rule_pool_cn policy_retriever/policy_pool/policy_cn.jsonl \ --log_model_io \ --log_dir model_deployment/logs ``` The input path must be a directory containing JSON files. Inputs already matching the MATE schema skip trajectory conversion but still pass through policy retrieval. ### 10. Troubleshooting **The benchmark smoke test reports missing directories** Populate `MateBench/MateBench-In`, `MateBench/MateBench-Out`, and `MateBench/MateBench-Real` with the released JSON files before running `run_eval_subset.sh`. **vLLM runs out of GPU memory** Start with MATE-0.5B and reduce context length or memory utilization: ```bash GPU_UTIL=0.7 \ MAX_LEN=4096 \ bash evaluation/launch_mate_vllm.sh Mate_model/MATE-0.5B MATE-0.5B 8016 ``` **Evaluation is too slow** Reduce request concurrency: ```bash python evaluation/run_matebench.py \ --task in \ --lang en \ --batch-size 8 \ --model MATE-1.5B ``` **The evaluator cannot connect to the model** ```bash curl http://127.0.0.1:8016/v1/models ``` Use `127.0.0.1` for a client on the same machine. Although vLLM may bind to `0.0.0.0`, clients should not use `0.0.0.0` as the destination address. **The retriever cannot find its model** Confirm that `policy_retriever/model/policy_retriever/` contains a valid SentenceTransformers model and that both policy pools exist: ```bash ls policy_retriever/model/policy_retriever ls policy_retriever/policy_pool/policy_{cn,en}.jsonl ``` **Offline rationale evaluation fails** Pre-download `BAAI/bge-large-zh-v1.5` and configure `evaluation/matebench_eval.py` to use its local path. ## Citation ```bibtex @misc{jiang2026mate, title={MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning}, author={Jiang, Changyue and Wang, Jiayi and Wen, Xin and Dai, Jiarun and Hong, Geng and Pan, Xudong}, year={2026} } ```