V2 Transfer & Imperceptibility Research Brief
Date: 2026-07-06
Context: White-box ensemble momentum-PGD on vision encoders for caption misread transfer. V1 at ε=12–16/255 achieves 100% in-ensemble, 66–75% held-out open encoders, 35–55% frontier (gpt-5.5 / gemini-3.5-flash) after JPEG — but PSNR 26–28 dB is visibly noisy. V2 must improve black-box transfer and genuine imperceptibility.
Papers cited below are web-verified (2023–2026 SOTA prioritized).
1. Black-Box Transfer Techniques
Mechanism = one line. Gain = typical black-box lift vs MI-FGSM / plain ensemble (ImageNet or VLM benchmarks). Fit = compatibility with our stochastic multi-encoder PGD + EOT loop.
Ranked shortlist (adopt → defer)
| Rank | Technique | Mechanism | Expected gain | Ensemble-PGD fit | Impl. cost |
|---|---|---|---|---|---|
| 1 | CWA (Common Weakness Attack) | SAM finds sharp ascent direction for flatter loss landscape; CSE aligns per-model gradients toward shared optima (ICLR 2024, code) | +5–15 pp vs MI-ensemble on diverse victims; demonstrated on Google Bard VLM | Excellent — replaces plain gradient sum across sampled encoders; orthogonal to EOT | Medium (~150 LOC in attack.py) |
| 2 | VNI-FGSM + VMI (variance tuning + Nesterov) | Lookahead position + subtract gradient variance over neighborhood samples to stabilize direction (arXiv:2103.15571) | +25–30 pp vs NI on adversarially trained CNNs; +5–10 pp typical on normal models | Excellent — drop-in replacement for MI-FGSM sign step | Low–medium |
| 3 | SVRE (Stochastic Variance Reduced Ensemble) | Outer-loop averaged gradient + inner-loop variance-reduced update prevents ensemble overfitting (CVPR 2022) | +3–8 pp vs uniform ensemble average | Excellent — we already do random subset; SVRE formalizes the outer/inner structure | Medium |
| 4 | Admix + DI + TI (input diversity stack) | Admix blends other-class patches; DI random resize/pad; TI convolves gradient with Gaussian kernel for translation invariance (ICCV 2021 Admix, CVPR 2019 TIM, CVPR 2019 DIM) | Admix-TI-DIM +3–6 pp over SI-TI-DIM on defenses; DIM alone ~+7 pp in NIPS 2017 comp | Good — apply transforms before encoder forward inside EOT; adds ~1.3× step time | Medium |
| 5 | Cross-prompt / CroPA-lite | Optimize image perturbation against a bank of caption prompts for the decoy so attack survives prompt diversity at inference (arXiv:2403.09766 CroPA, GrCPA NeurIPS 2024 submission) | CroPA reports strong cross-prompt ASR on BLIP-2/LLaVA; GrCPA stabilizes multi-prompt gradients | Good — extend contrastive loss: mean_p L(cos(f, e_{decoy}(p))); no full VLM backward needed |
Low–medium |
| 6 | CoA-style multimodal semantic chain (lite) | Iteratively refine decoy text embedding from surrogate caption of current adv image, then re-attack toward modality-aware target (CVPR 2025 CoA) | CoA +1–7 pp over MF-ii on CLIP variants; 75.2% CLIP-score vs 70% baselines | Partial — full CoA needs ClipCap loop; lite = refresh decoy text every 50 steps via frozen BLIP/CLIP prefix | High (full) / Medium (lite) |
| 7 | Margin-calibrated embedding loss (feature > raw CE) | Normalized logit / cosine margin with adaptive temperature avoids vanishing gradients in targeted transfer (NeurIPS 2021 Logit, TIFS 2023 Logit Margin, arXiv:2405.06340 NLCM+TFM) | Logit loss +10–20 pp vs CE in hard ensemble transfer; feature mixing +5–10 pp | Excellent — we already use cosine margin; add margin temperature + truncated feature mix on repel term | Low |
| 8 | SSA (spectrum simulation) | DCT-domain random mask simulates diverse spectral surrogate models (ECCV 2022) | Part of SOTA combo attacks; ~+3–5 pp stacked with DIM/TIM | Good — frequency aug parallel to spatial EOT; cheap with torch.fft or DCT |
Medium |
| 9 | Ghost networks | On-the-fly dropout / skip-connection noise creates virtual model ensemble from each encoder (AAAI 2020) | Effective multiplier on ensemble size at ~0 extra model load | Good — apply to 2–3 largest ViT encoders per step | Low |
| 10 | SEA-FTS (fine-tuning trajectory simulation) | Random weight noise in vision encoder simulates fine-tuned VLM variants during optimization (arXiv:2508.01741) | >86% jailbreak transfer across Qwen2-VL fine-tunes vs near-0% plain PGD | Partial — needs differentiable encoder weights; works on timm ViTs, not frozen CLIP text towers | Medium–high |
| 11 | X-Transfer surrogate scaling | Dynamic selection of surrogate CLIP encoders from large pool for UAP/super-transfer (ICML 2025) | SOTA on cross-model CLIP + captioning transfer | Partial — we already do random subset; upgrade to gradient-weighted or diversity-scored selection | Low |
| 12 | SGMA (semantic-guided multimodal attack) | Perturb semantically salient regions; disrupt global/local cross-modal grounding (arXiv:2602.09431) | Best encoder-only LVLM transfer in 2026 benchmark | Partial — needs saliency maps from each encoder; pairs with repel loss | High |
| 13 | EMI-FGSM | Accumulates gradients along previous momentum direction samples (arXiv:2103.10609) | +10–12 pp vs PI-FGSM | Redundant with VMI — pick one | Low |
| 14 | SIM / NI alone | Scale copies / Nesterov lookahead | Foundational; subsumed by VNI+SI stack | Already partially via EOT resize | — |
| 15 | AttackVLM query (RGF) | Transfer init + black-box score queries refine caption match (AttackVLM) | Large lift with API queries | Different threat model — defer to Tier-B frontier polish | High + $$ |
| 16 | UnivIntruder / X-Transfer UAP | Universal perturbation across images | Great for UAP, not per-image training data protection | Poor fit for per-image targeted decoy | — |
VLM-/CLIP-specific notes
- Feature-space attacks transfer better than logit/CE for CLIP when the victim uses a different head: embedding cosine margin (our v1) is correct; improve with normalized margin calibration (NLCM) and rank-1 truncated feature mixing on the repel branch.
- Cross-prompt is mandatory for caption victims: frontier models use free-form prompts; a perturbation optimized for one decoy string fails when the API uses a different template. CroPA/GrCPA directly address this (CroPA).
- CoA / AttackVLM confirm that target image + target text joint embedding beats image-only attacks for caption transfer (CoA, AttackVLM) — we should add a frozen decoy-image embedding (SD/DALL·E stills or CLIP text-only) alongside text decoy.
- Encoder-only attacks remain the right efficiency tradeoff; SGMA shows encoder attacks can transfer to full LVLMs if semantics are targeted, but need better grounding than v1 repel loss.
Algorithm sketch — transfer stack (per PGD step)
δ ← 0; ḡ ← 0; m ← 0; v ← 0 # SVRE outer, VNI momentum, optional Adam v
for t = 1..T:
# NI lookahead
x_la = clamp(x0 + δ + α·m, 0, 1)
g_list = []
for k = 1..K_inner: # SVRE inner
S ← sample_encoders(pool, n=6)
x_in ← Admix(DI(x_la)) # p_di=0.7, mix_ratio=0.2
for j = 1..N_eot: # N_eot=3
x_t ← EOT(x_in) # DiffJPEG curriculum + resize + blur
for enc in S:
g_list.append(∇_δ L_enc(x_t)) # margin-calibrated cosine loss
g ← CSE(SAM_aggregate(g_list)) # CWA
g ← VMI_tune(g, history) # variance from N=20 gaussian δ jitter
m ← μ·m + g/||g||_1 # VNI momentum
δ ← δ + α · sign(m)
δ ← Π_{L∞}(δ; ε=8/255)
δ ← Π_{LPIPS}(δ; τ=0.08) # Section 2
δ ← Π_{DCT-low}(δ) # Section 2
Suggested hyperparameters (transfer)
| Param | Value | Source rationale |
|---|---|---|
| Steps T | 300–400 | More steps compensate lower ε |
| ε∞ | 8/255 (down from 12–16) | Imperceptibility budget; stack transfer methods instead of raw ε |
| α (step) | 1.0/255 | Standard for ε=8 |
| μ (momentum) | 1.0 | TIM/MI default |
| NI decay | 1.0 | NI-FGSM |
| VMI neighbors N | 20, σ=1.5/255 | VMI |
| SAM ρ | 0.05·ε | CWA |
| Subset size | 6 (of 12–14) | v1 used 6; SVRE benefits from ≥4 |
| DI prob p | 0.7 | DIM tradeoff: high p → transfer, mild white-box hit |
| TI kernel | 15×15 Gaussian | TIM |
| Admix ratio | 0.2, 3 add-in crops | Admix |
| Prompt bank | 8–16 paraphrases of decoy caption | CroPA |
| Margin temp | 2.5–5.0 adaptive | Logit Margin |
Expected transfer benefit (v2 transfer stack alone): frontier 35–55% → 50–65% post-JPEG at matched perceptibility; held-out open 66–75% → 75–85%. Gains are sublinear — budget A/B each component.
2. Perceptual / Imperceptible Constraints Beyond L∞
L∞ with ε=12–16/255 yields PSNR 26–28 dB in v1 — visible high-frequency speckle. Human perception correlates better with LPIPS (ρ≈0.94) than L2 (ρ≈0.88) (ICLR 2021 Perceptual Adversarial Robustness).
Ranked shortlist
| Rank | Constraint | Mechanism | Imperceptibility vs strength | PGD implementation | Impl. cost |
|---|---|---|---|---|---|
| 1 | Dual L∞ + LPIPS bound (PPGD / LPA) | Project (or penalize) so LPIPS(x, x_adv) ≤ τ in addition to ε∞ (ICLR 2021, LPA strongest) |
Best human-imperceptible attack strength at fixed τ; finds attacks outside Lp balls | Penalty: L += λ·max(0, LPIPS−τ); Projection: scale δ after L∞ clip (binary search or 3 Newton steps) |
Medium (lpips lib) |
| 2 | Texture-masked adaptive ε map (JND / local variance) | Larger ε in high-texture, smaller on edges/smooth regions (arXiv:2011.05254, ICCV 2025 IAP) | Same L∞ norm but looks cleaner; exploits HVS masking | ε(i,j) = ε_base · (1 + β·σ_local(i,j)), normalize max to ε_base; multiply gradient step |
Low |
| 3 | DCT low-frequency subspace | Zero high DCT coefficients; perturbation is smooth/low-freq (IJCAI 2019, LFAA 2023, FDT 2024) | Much less visible speckle; bonus: survives JPEG better (energy in kept bands) | δ ← IDCT(M_low ⊙ DCT(δ)) each step; M_low keeps coeffs with freq < r (r=H/8) |
Low |
| 4 | Lagrangian joint objective (attack + λ·LPIPS) | LPA-style: stronger than pure projection at same τ (ICLR 2021) | Best attack success per perceptual unit | λ annealed: start 0.1, increase if LPIPS<τ/2 at end | Medium |
| 5 | CIEDE2000 (ΔE00) cap | Perceptual color distance in Lab/LCH (arXiv:1911.02466 PerC) | Superior on smooth/saturated regions (sky, walls) where L∞ speckle shows | Differentiable approx of ΔE00; project in Lab space after RGB step | Medium |
| 6 | Chrominance-only (CbCr / ab) | Restrict δ to chroma channels (arXiv:2310.13950) | Very subtle on natural photos; weaker on grayscale objects | RGB→YCbCr, zero δ_Y; or optimize in Lab ab only | Low |
| 7 | SSIM / MS-SSIM ball | Convex SSIM constraint (IEEE JSTSP 2023) | Good semantics (changes in salient regions) but weaker correlation than LPIPS | Penalty on 1−SSIM; pairs with project gate |
Low |
| 8 | Unrestricted / semantic (color shift, warp) | Not L∞-bounded; different threat model | Most imperceptible per unit effect; poor JPEG compatibility for warp | Defer — training data protection needs pixel-stable JPEG survival | — |
Best tradeoff for our use case
Recommended combo: ε∞=8/255 hard cap + LPIPS ≤ 0.08 (AlexNet backbone, ImageNet-scale) + DCT low-pass (r≤H/8) + texture mask β=2.0.
Rationale:
- LPIPS aligns with our project stealth gate (LPIPS ≤ 0.10 strict) in
PROJECT_MEMORY.md. - DCT-lowpass removes speckle that PSNR misses but humans see; also aligns JPEG-survival literature (MetaCloak-JPEG notes high-freq energy is wiped by quant tables).
- Texture mask concentrates budget where v1 noise is already less visible.
Differentiable implementation recipes
LPIPS projection (after gradient step):
# δ' = argmin_{||δ||∞≤ε} ||δ|| s.t. LPIPS(x0, x0+δ) ≤ τ
# Practical: if LPIPS > τ, δ ← δ * (τ / LPIPS)^γ with γ≈1.5, repeat 3×
LPIPS penalty (inside loss):
L_total = L_attack + λ_lp * relu(lpips(x_adv, x0) - τ)
# λ_lp = 10–50; anneal down last 20% steps to recover margin
DCT mask:
# Use orthonormal DCT-II on 8×8 blocks (JPEG-compatible)
delta_low = idct8x8(dct8x8(delta) * low_freq_mask) # mask top-left k×k coeffs
Texture mask:
# σ from 7×7 local std on grayscale x0
eps_map = eps_base * (1 + beta * (sigma / sigma.max()))
delta = clip(delta, -eps_map, eps_map)
Suggested perceptual hyperparameters
| Param | Strict (deploy) | Medium (research) |
|---|---|---|
| ε∞ | 6/255 | 8/255 |
| LPIPS (Alex) τ | 0.06 | 0.08 |
| DCT keep ratio | r = H/16 | r = H/8 |
| Texture β | 1.5 | 2.0 |
| ΔE00 p95 cap | 2.0 (text-adjacent regions) | 3.0 |
| Target PSNR | ≥ 38 dB | ≥ 34 dB |
| Target SSIM | ≥ 0.94 | ≥ 0.90 |
Expected benefit: PSNR 26–28 → 34–40 dB, SSIM 0.65–0.74 → 0.90–0.95, with ~15–25% relative drop in raw margin unless compensated by longer optimization + transfer stack.
3. JPEG / Resize / Crop Robustness (EOT Design)
V1 uses BPDA straight-through JPEG + probabilistic resize/blur/crop in ensemble/eot.py. Margins survive JPEG (+0.195 vs +0.016 pre-EOT) — EOT works. V2 should improve gradient fidelity and QF coverage without destroying transfer.
JPEG: differentiable approx vs BPDA
| Method | Mechanism | Pros | Cons |
|---|---|---|---|
| BPDA / STE (v1) | Forward: real PIL JPEG; backward: identity (MLSec 2017) | Simple, fast, any QF | Gradient ignores quant drop; HF energy falsely encouraged |
| DiffJPEG + STE on round | Full YCbCr→DCT→quant with STE backward (WACV 2024) | Gradients through quant; better HF survival under JPEG | Heavier; must match libjpeg tables |
| MetaCloak-JPEG curriculum | QF sampled from [QF_min(t), 95] rising over steps; ~70% aug include JPEG (arXiv:2604.18537) |
91% JPEG survival vs baseline; PSNR 32.7 dB at ε=8 | Tuned for diffusion; principles transfer |
Recommendation: Hybrid — default to DiffJPEG-STE for backward; keep PIL JPEG in 30% of EOT samples as "reality check" (no grad) to avoid overfitting differentiable approx.
EOT transform menu (ranked)
| Priority | Transform | Sample prob | Param range | Role |
|---|---|---|---|---|
| 1 | JPEG | 0.70 | QF ∈ {45,55,65,75,85,92}, curriculum min 45→75 | Primary threat |
| 2 | Resize round-trip | 0.55 | scale ∈ [0.5, 1.0] | Scraper downscale |
| 3 | Gaussian blur | 0.35 | σ ∈ [0.3, 1.2] | CDN / recompress |
| 4 | Center/random crop | 0.25 | keep ∈ [0.88, 0.98] | Thumbnail crop |
| 5 | Gamma / brightness | 0.15 | ±8% | Light EOT diversity |
| 6 | SSA spectral mask | 0.20 | (transfer, Section 1) | Model diversity |
How many EOT samples / step?
| N_eot | Robustness | Transfer | Compute |
|---|---|---|---|
| 1 (v1) | Fragile to QF mismatch | Baseline | 1× |
| 3 (v2 rec) | Good QF ensemble | +3–5 pp | 3× |
| 5+ | Diminishing returns | slight white-box hit | 5×+ |
Literature consensus: Ensemble multiple QFs with loss-weighted gradients (MLSec 2017); EOT sample count 3–5 (EOT original); more transforms require larger perceptual budget or attack fails to converge (DIM p tradeoff).
Robustness vs transfer tradeoff
More aggressive EOT ──► better JPEG survival, better black-box transfer
──► needs more steps / milder ε or LPIPS absorbs slack
──► white-box margin slower (acceptable: we care about held-out)
Practical split (v2):
- 70% of steps:
EOT_strong(JPEG+resize+blur) - 30% of steps:
EOT_mild(resize only) — preserves white-box convergence signal
Algorithm sketch — EOT module v2
def eot_v2(x, t, rng):
# Curriculum: QF_min rises 45 → 75 over t/T
qf_lo = int(45 + 30 * t / T)
if rng.random() < 0.70:
qf = rng.choice(range(qf_lo, 96, 5))
x = diffjpeg_ste(x, qf) # WACV 2024 backward
if rng.random() < 0.30:
x = jpeg_ste_pil(x, qf) # reality check, no grad
if rng.random() < 0.55:
x = resize_roundtrip(x, rng.uniform(0.5, 1.0))
if rng.random() < 0.35:
x = gaussian_blur(x, rng.uniform(0.3, 1.2))
if rng.random() < 0.25:
x = random_crop_pad(x, rng.uniform(0.88, 0.98))
return x.clamp(0, 1)
Expected benefit: held-out post-JPEG margin variance ↓ 30–50%; frontier +3–8 pp from better QF generalization without raising ε.
Implementation cost: DiffJPEG integration medium (1–2 days); curriculum + multi-sample EOT low.
4. Recommended V2 Attack Recipe
Name: Ensemble-SVRE-CWA-VNI-Perceptual-EOT (ESCVPE)
Single combined recipe — implement in phases P0→P2.
Phase P0 (1–2 days) — highest ROI
- Drop ε∞ to 8/255; add LPIPS projection τ=0.08.
- Replace MI with VNI + VMI (N=20).
- Add CWA gradient aggregation across sampled encoders.
- Cross-prompt bank (8 paraphrases) on contrastive decoy loss.
- EOT: N_eot=3, JPEG QF ensemble {55,75,85}.
Phase P1 (2–3 days) — transfer lift
- SVRE outer/inner loop over encoder subsets.
- Admix + DI + TI input stack.
- DCT low-pass mask (r=H/8).
- Texture-masked ε map (β=2).
- Add 3–4 VLM vision towers (AIMv2, DINOv2, SigLIP2-NaFlex, InternViT) — already de-risked in v1 ablation.
Phase P2 (3–5 days) — frontier polish
- DiffJPEG-STE replacing pure BPDA.
- Margin temperature calibration + truncated feature mixing on repel loss.
- Ghost noise on largest ViT encoders (p_drop=0.1).
- CoA-lite: refresh decoy text embedding every 50 steps from CLIP-prefix caption of
x_adv. - Optional SEA-FTS noise on timm encoders (σ_w=0.01).
Loss function (v2)
L = Σ_enc w_enc · (
w_t · mean_p [ margin_calib( cos(f(x), e_decoy(p)), cos(f(x), e_truth) ) ]
+ w_r · [ -cos(f(x), f_clean) + mix_truncated_features(f, f_clean) ]
+ w_i · [ cos(f(x), e_decoy_img) - cos(f(x), e_truth) ] # optional decoy image embed
)
Full hyperparameter card
eps_inf: 0.03137 # 8/255
lpips_tau: 0.08
dct_keep: 0.125 # H/8
texture_beta: 2.0
steps: 350
step_size: 0.00392 # 1/255
momentum: 1.0
vni: true
vmi_neighbors: 20
cwa_sam_rho: 0.05
svre_inner: 3
encoders_per_step: 6
encoder_pool: 12-14
eot_samples: 3
eot_jpeg_prob: 0.70
prompt_bank: 12
w_target: 1.0
w_repel: 0.5
w_decoy_img: 0.3
margin_temp: 3.0
Expected end-state (vs v1 @ ε=12–16)
| Metric | V1 | V2 target |
|---|---|---|
| PSNR | 26–28 dB | ≥ 36 dB |
| SSIM | 0.65–0.74 | ≥ 0.92 |
| LPIPS | ~0.15–0.25 (est.) | ≤ 0.08 |
| Held-out open flip (JPEG) | 66–75% | 75–85% |
| Frontier caption flip | 35–55% | 50–65% |
| White-box in-ensemble | 100% | 95–100% (acceptable dip) |
| Step time | 1× | ~3–4× |
Implementation priority / cost summary
| Component | Cost | Priority |
|---|---|---|
| VNI+VMI+CWA | 1 d | P0 |
| LPIPS dual constraint | 1 d | P0 |
| Cross-prompt bank | 0.5 d | P0 |
| EOT multi-sample + QF ensemble | 0.5 d | P0 |
| SVRE + Admix/DI/TI | 2 d | P1 |
| DCT + texture mask | 1 d | P1 |
| DiffJPEG-STE | 2 d | P2 |
| CoA-lite / decoy image embed | 2 d | P2 |
| Ghost + SEA-FTS | 1 d | P2 |
References (verified URLs)
Document written for veil-pgd v2 planning. Next: implement P0 in ensemble/attack.py, sweep ε∈{4,6,8} × LPIPS τ∈{0.06,0.08,0.10} on Imagenette, re-run frontier_eval.