Testing plan
What a "win" means
Untargeted attack: make the VLM's label wrong and far from the truth. We do NOT require forcing a specific decoy. The metric is semantic distance of the model's returned label from the true label.
A trial is a success only if all three hold:
- semantic distance to truth >= tau (report tau in {0.3, 0.5, 0.7}),
- the render passed the stealth gate at that level, and
- it still succeeds after
scraper_sim(JPEG Q85 + light blur).
Reporting strength without stealth (or vice versa) is misleading, so we always report the pair.
Targets
- Black-box:
openai/gpt-5.5andgoogle/gemini-3.5-flash(OpenRouter), the two strongest vision models. Report per-model and the transfer between them. - Surrogates (free Tier A search):
qwen-3.5-4b+gemma-4-4bon klaus-3.
Image set (standard, comparable to literature)
~30-50 images with human-verified one-word truths, from a standard source (ImageNet/COCO-style), spanning a difficulty gradient:
- ambiguous / cluttered scenes (easiest to poison),
- ordinary single-subject photos (the realistic use case),
- iconic / unambiguous subjects (the hard ceiling; keeps us honest).
CSV format for the harness: image_path,truth_label per line.
Stealth sweep (the headline experiment)
Run every image at three gate levels and plot the strength-vs-visibility
frontier (veil-pgd sweep images.csv):
| level | PSNR>= | SSIM>= | LPIPS<= | dE2000 p95<= |
|---|---|---|---|---|
| strict | 38 | 0.94 | 0.10 | 2.0 |
| medium | 32 | 0.90 | 0.20 | 5.0 |
| loose | 26 | 0.82 | 0.40 | 12.0 |
Expected shape: success rate rises as stealth loosens. The literature's ~41-64% band is for faint-but-typographic (roughly our strict/medium).
Metrics reported
- poison success rate at each tau, per stealth level;
- distance-to-truth CDF;
- per-model success + transfer matrix (gpt-5.5 vs gemini-3.5-flash);
- stealth pass rate per level;
- robustness delta: success on clean render vs after JPEG/blur, plus the
hard_transformsceiling (heavy blur, downscale, rotation) for context.
Known robustness ceiling
OCR-stripping / Dyslexify-style text removal, heavy blur, aggressive downscale, and rotation can neutralize the overlay. These are reported, not defended against; the tool raises scraping cost, it is not permanent protection.