README.md

---
title: SamplingTAR Defense
emoji: 🛡️
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 6.19.0
app_file: app.py
short_description: Training-free typographic attack defense for CLIP
python_version: "3.12"
startup_duration_timeout: 30m
---

# SamplingTAR: Training-free Defense against Typographic Attacks

Interactive demo of the training-free concept localization method from
[Towards Robustness against Typographic Attack with Training-free Concept Localization](https://huggingface.co/papers/2607.02494).

## How it works

Typographic attacks overlay text on images to fool vision-language models like CLIP
into predicting the text rather than the actual object. SamplingTAR defends against
these attacks **without any training** by:

1. **Mining text-reading heads** — Randomly-initialised Sparse Autoencoders (SAEs)
   probe which attention heads in CLIP's vision transformer localise to text regions.
2. **Ablating text heads** — At inference, the CLS-token attention from those heads
   to text patches is redistributed, neutralising the typographic attack.

## Usage

1. Upload an image (or use an example).
2. Enter candidate labels (comma-separated).
3. Click "Run Defense" to compare undefended vs. defended predictions and view
   attention heatmaps.