docs/TESTING.md

# Testing plan

## What a "win" means
Untargeted attack: make the VLM's label **wrong and far from the truth**. We do
NOT require forcing a specific decoy. The metric is semantic distance of the
model's returned label from the true label.

A trial is a **success** only if all three hold:
1. semantic distance to truth >= tau (report tau in {0.3, 0.5, 0.7}),
2. the render passed the stealth gate at that level, and
3. it still succeeds after `scraper_sim` (JPEG Q85 + light blur).

Reporting strength without stealth (or vice versa) is misleading, so we always
report the pair.

## Targets
- Black-box: `openai/gpt-5.5` and `google/gemini-3.5-flash` (OpenRouter), the two
  strongest vision models. Report per-model and the transfer between them.
- Surrogates (free Tier A search): `qwen-3.5-4b` + `gemma-4-4b` on klaus-3.

## Image set (standard, comparable to literature)
~30-50 images with human-verified one-word truths, from a standard source
(ImageNet/COCO-style), spanning a difficulty gradient:
- ambiguous / cluttered scenes (easiest to poison),
- ordinary single-subject photos (the realistic use case),
- iconic / unambiguous subjects (the hard ceiling; keeps us honest).

CSV format for the harness: `image_path,truth_label` per line.

## Stealth sweep (the headline experiment)
Run every image at three gate levels and plot the strength-vs-visibility
frontier (`veil-pgd sweep images.csv`):

| level  | PSNR>= | SSIM>= | LPIPS<= | dE2000 p95<= |
|--------|--------|--------|---------|--------------|
| strict | 38     | 0.94   | 0.10    | 2.0          |
| medium | 32     | 0.90   | 0.20    | 5.0          |
| loose  | 26     | 0.82   | 0.40    | 12.0         |

Expected shape: success rate rises as stealth loosens. The literature's
~41-64% band is for faint-but-typographic (roughly our strict/medium).

## Metrics reported
- poison success rate at each tau, per stealth level;
- distance-to-truth CDF;
- per-model success + transfer matrix (gpt-5.5 vs gemini-3.5-flash);
- stealth pass rate per level;
- robustness delta: success on clean render vs after JPEG/blur, plus the
  `hard_transforms` ceiling (heavy blur, downscale, rotation) for context.

## Known robustness ceiling
OCR-stripping / Dyslexify-style text removal, heavy blur, aggressive downscale,
and rotation can neutralize the overlay. These are reported, not defended
against; the tool raises scraping cost, it is not permanent protection.