veil-pgd / docs /TESTING.md
Klaus Clawd
Initial public release: VEIL-PGD v0.1
c793f45
|
Raw
History Blame Contribute Delete
2.42 kB

Testing plan

What a "win" means

Untargeted attack: make the VLM's label wrong and far from the truth. We do NOT require forcing a specific decoy. The metric is semantic distance of the model's returned label from the true label.

A trial is a success only if all three hold:

  1. semantic distance to truth >= tau (report tau in {0.3, 0.5, 0.7}),
  2. the render passed the stealth gate at that level, and
  3. it still succeeds after scraper_sim (JPEG Q85 + light blur).

Reporting strength without stealth (or vice versa) is misleading, so we always report the pair.

Targets

  • Black-box: openai/gpt-5.5 and google/gemini-3.5-flash (OpenRouter), the two strongest vision models. Report per-model and the transfer between them.
  • Surrogates (free Tier A search): qwen-3.5-4b + gemma-4-4b on klaus-3.

Image set (standard, comparable to literature)

~30-50 images with human-verified one-word truths, from a standard source (ImageNet/COCO-style), spanning a difficulty gradient:

  • ambiguous / cluttered scenes (easiest to poison),
  • ordinary single-subject photos (the realistic use case),
  • iconic / unambiguous subjects (the hard ceiling; keeps us honest).

CSV format for the harness: image_path,truth_label per line.

Stealth sweep (the headline experiment)

Run every image at three gate levels and plot the strength-vs-visibility frontier (veil-pgd sweep images.csv):

level PSNR>= SSIM>= LPIPS<= dE2000 p95<=
strict 38 0.94 0.10 2.0
medium 32 0.90 0.20 5.0
loose 26 0.82 0.40 12.0

Expected shape: success rate rises as stealth loosens. The literature's ~41-64% band is for faint-but-typographic (roughly our strict/medium).

Metrics reported

  • poison success rate at each tau, per stealth level;
  • distance-to-truth CDF;
  • per-model success + transfer matrix (gpt-5.5 vs gemini-3.5-flash);
  • stealth pass rate per level;
  • robustness delta: success on clean render vs after JPEG/blur, plus the hard_transforms ceiling (heavy blur, downscale, rotation) for context.

Known robustness ceiling

OCR-stripping / Dyslexify-style text removal, heavy blur, aggressive downscale, and rotation can neutralize the overlay. These are reported, not defended against; the tool raises scraping cost, it is not permanent protection.