README.md

metadata

title: SamplingTAR Defense
emoji: 🛡️
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 6.19.0
app_file: app.py
short_description: Training-free typographic attack defense for CLIP
python_version: '3.12'
startup_duration_timeout: 30m

SamplingTAR: Training-free Defense against Typographic Attacks

Interactive demo of the training-free concept localization method from Towards Robustness against Typographic Attack with Training-free Concept Localization.

How it works

Typographic attacks overlay text on images to fool vision-language models like CLIP into predicting the text rather than the actual object. SamplingTAR defends against these attacks without any training by:

1. Mining text-reading heads — Randomly-initialised Sparse Autoencoders (SAEs) probe which attention heads in CLIP's vision transformer localise to text regions.
2. Ablating text heads — At inference, the CLS-token attention from those heads to text patches is redistributed, neutralising the typographic attack.

Usage

1. Upload an image (or use an example).
2. Enter candidate labels (comma-separated).
3. Click "Run Defense" to compare undefended vs. defended predictions and view attention heatmaps.