File size: 1,662 Bytes
555da34 bfe0177 2f856d4 37ee7ea 555da34 bfe0177 555da34 2f856d4 555da34 2f856d4 555da34 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | ---
datasets:
- Jarvis1111/RobustVLGuard
license: mit
pipeline_tag: image-text-to-text
library_name: transformers
base_model:
- OpenGVLab/InternVL2-8B
---
# 🚀 Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks
Welcome! This repository hosts the official implementation of our paper, **"Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks."**
Paper link: arxiv.org/abs/2504.01308
---
## 🌟 What’s New?
We propose state-of-the-art solutions to enhance the robustness of Vision-Language Models (VLMs) against Gaussian noise and adversarial attacks. Key highlights include:
- 🎯 **Robust-VLGuard**: A pioneering multimodal safety dataset covering both aligned and misaligned image-text pair scenarios.
- 🛡️ **DiffPure-VLM**: A novel defense framework that leverages diffusion models to neutralize adversarial noise by transforming it into Gaussian-like noise, significantly improving VLM resilience.
---
## ✨ Key Contributions
- 🔍 Conducted a comprehensive vulnerability analysis revealing the sensitivity of mainstream VLMs to Gaussian noise.
- 📚 Developed **Robust-VLGuard**, a dataset designed to improve model robustness without compromising helpfulness or safety alignment.
- ⚙️ Introduced **DiffPure-VLM**, an effective pipeline for defending against complex optimization-based adversarial attacks.
- 📈 Demonstrated strong performance across multiple benchmarks, outperforming existing baseline methods.
--- |