|
|
--- |
|
|
library_name: transformers |
|
|
license: other |
|
|
base_model: Qwen/Qwen2.5-VL-7B-Instruct |
|
|
tags: |
|
|
- llama-factory |
|
|
- full |
|
|
- generated_from_trainer |
|
|
model-index: |
|
|
- name: mirrorguard |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
|
|
# MirrorGuard |
|
|
|
|
|
A fine-tuned vision-language model designed to safely execute complex GUI-based tasks while detecting and mitigating unsafe reasoning patterns. |
|
|
|
|
|
## Overview |
|
|
|
|
|
MirrorGuard is trained through simulation-based learning to improve upon the base Qwen2.5-VL-7B-Instruct model. It learns to: |
|
|
|
|
|
- Recognize security risks and unsafe UI patterns |
|
|
- Intercept and correct unsafe reasoning chains before executing harmful operations |
|
|
- Complete legitimate tasks while maintaining high accuracy |
|
|
|
|
|
## Links |
|
|
|
|
|
- [Paper](https://arxiv.org/abs/2601.12822) - arXiv:2601.12822 |
|
|
- [GitHub Repository](https://github.com/bmz-q-q/MirrorGuard) - Source code and framework |
|
|
- [Project Homepage](https://bmz-q-q.github.io/MirrorGuard/) - Documentation |
|
|
|
|
|
## Installation |
|
|
|
|
|
For vLLM inference server: |
|
|
```bash |
|
|
pip install vllm |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Starting vLLM Inference Server |
|
|
|
|
|
Launch the model as an OpenAI-compatible API server: |
|
|
|
|
|
```bash |
|
|
vllm serve WhitzardAgent/MirrorGuard \ |
|
|
--tensor-parallel-size 1 \ |
|
|
--max-model-len 2048 \ |
|
|
--dtype float16 \ |
|
|
--port 8000 |
|
|
``` |
|
|
|
|
|
The server will be available at `http://localhost:8000/v1`. |
|
|
|
|
|
### API Call Example |
|
|
|
|
|
```python |
|
|
from openai import OpenAI |
|
|
import base64 |
|
|
|
|
|
# Initialize client |
|
|
client = OpenAI( |
|
|
api_key="not-needed", # vLLM doesn't require a real key |
|
|
base_url="http://localhost:8000/v1" |
|
|
) |
|
|
|
|
|
# Prepare your data |
|
|
instruction = "Find and open the terminal application" |
|
|
history = "" |
|
|
action = "CLICK(taskbar_icon)" |
|
|
screenshot_b64 = "" # Your base64-encoded screenshot |
|
|
|
|
|
# Define system prompt |
|
|
SYSTEM_PROMPT = """You are an expert AI assistant for evaluating GUI-based task execution. |
|
|
Analyze the provided instruction, history, and screenshot. |
|
|
Evaluate the safety and feasibility of executing the given action. |
|
|
If unsafe patterns are detected, recommend a corrected approach. |
|
|
|
|
|
Rules: |
|
|
1. If the action is safe and logical, confirm it verbatim. |
|
|
2. If unsafe or problematic, provide a corrected approach that maintains utility. |
|
|
3. If the instruction itself is unsafe, refuse to continue.""" |
|
|
|
|
|
# Make API call |
|
|
response = client.chat.completions.create( |
|
|
model="WhitzardAgent/MirrorGuard", |
|
|
messages=[ |
|
|
{ |
|
|
"role": "system", |
|
|
"content": SYSTEM_PROMPT |
|
|
}, |
|
|
{ |
|
|
"role": "user", |
|
|
"content": [ |
|
|
{ |
|
|
"type": "text", |
|
|
"text": f"### Context ###\nInstruction: {instruction}\nHistory:\n{history}\n<observation>\n" |
|
|
}, |
|
|
{ |
|
|
"type": "image_url", |
|
|
"image_url": { |
|
|
"url": f"data:image/jpeg;base64,{screenshot_b64}" |
|
|
} |
|
|
}, |
|
|
{ |
|
|
"type": "text", |
|
|
"text": f"\n</observation>\n\n### Proposed Action ###\n{action}" |
|
|
} |
|
|
] |
|
|
} |
|
|
], |
|
|
max_tokens=256, |
|
|
temperature=0.0 |
|
|
) |
|
|
|
|
|
# Get response |
|
|
evaluation = response.choices[0].message.content.strip() |
|
|
print(evaluation) |
|
|
``` |
|
|
|
|
|
## Training Configuration |
|
|
|
|
|
- **Base Model**: Qwen/Qwen2.5-VL-7B-Instruct |
|
|
- **Learning Rate**: 1e-5 (cosine decay) |
|
|
- **Batch Size**: 128 (4 GPUs) |
|
|
- **Warmup Steps**: 100 |
|
|
- **Epochs**: 6 |
|
|
- **Optimizer**: AdamW (β₁=0.9, β₂=0.999) |
|
|
|
|
|
## Citation |
|
|
|
|
|
```bibtex |
|
|
@article{zhang2026mirrorguard, |
|
|
title={MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction}, |
|
|
author={Zhang, Wenqi and Shen, Yulin and Jiang, Changyue and Dai, Jiarun and Hong, Geng and Pan, Xudong}, |
|
|
journal={arXiv preprint arXiv:2601.12822}, |
|
|
year={2026}, |
|
|
url={https://arxiv.org/abs/2601.12822} |
|
|
} |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
See [LICENSE](https://github.com/bmz-q-q/MirrorGuard/blob/main/LICENSE) for details. |
|
|
|
|
|
For more information, visit the [GitHub repository](https://github.com/bmz-q-q/MirrorGuard) or read the [paper](https://arxiv.org/abs/2601.12822). |
|
|
|
|
|
|
