| --- |
| license: other |
| license_name: polyform-noncommercial-1.0.0 |
| license_link: https://polyformproject.org/licenses/noncommercial/1.0.0 |
| tags: |
| - training-data-protection |
| - opt-out |
| - protective-perturbations |
| - adversarial-examples |
| - vision-language-models |
| - clip |
| - image-protection |
| - anti-scraping |
| - data-poisoning |
| - pgd |
| pretty_name: VEIL-PGD |
| --- |
| |
| # VEIL-PGD: Vision-Encoder Imperceptible Layer |
|
|
| *A research preview (v0.2.1) of an image-side protection method against unauthorized |
| vision and vision-language training-data ingestion.* |
|
|
| ## Abstract |
|
|
| Public images are routinely collected into web-scale datasets and used to train vision |
| and vision-language models, and there is rarely a practical way to opt out. We present |
| VEIL-PGD (Vision-Encoder Imperceptible Layer), a creator-controlled defense that adds a |
| low-visibility adversarial perturbation to an image before publication. The goal is to |
| make the supervision an ingestion pipeline extracts from that image, such as captions, |
| tags, retrieval embeddings, and synthetic labels, semantically displaced from the true |
| content. The perturbation is an Expectation-over-Transformation (EOT) hardened, ensemble, |
| momentum-PGD attack optimized against a diverse set of public vision encoders. That set |
| spans contrastive CLIP-family models, self-supervised and auto-supervised feature towers, |
| and the vision towers of recent open vision-language models. On a 60-image benchmark with |
| bootstrap confidence intervals, the perturbation causes frontier captioners (GPT-5.5 and |
| Gemini 3.5 Flash) to prefer a decoy label over the true label on 45.2% of the images they |
| originally labeled correctly, measured after JPEG recompression. Transfer to held-out |
| encoders that never took part in the attack, including a real VLM captioner tower and an |
| agglomerative foundation model, reaches 28.5%. This is arms-race mitigation rather than a |
| permanent or universal guarantee, and the results are early. This repository releases the |
| method and code, and it ships no trained weights. |
|
|
| ## 1. Introduction and motivation |
|
|
| For many public images there is no dependable way to opt out of dataset collection. |
| Platform controls are uneven, scraper behavior is opaque, and once an image has spread |
| through mirrors, indexes, reposts, and data brokers, policy-level signals are hard to |
| enforce. VEIL-PGD treats protection as something the image itself can carry. The creator |
| applies a perturbation before publication that degrades the image's usefulness as |
| training data while preserving normal human viewing. It is a harm-reduction layer, not a |
| replacement for consent or a legal opt-out. |
|
|
| ## 2. Threat model |
|
|
| - **Defender.** The image owner, who can modify an image once before publishing and has |
| no control over downstream pipelines. |
| - **Adversary.** An automated ingestion pipeline that derives weak supervision from |
| images at scale, such as auto-captioning and tagging, CLIP-style embedding and |
| retrieval filtering, VLM-based synthetic labeling, and aesthetic or safety scoring, and |
| then trains on the resulting image-text pairs or labels. |
| - **Adversary capabilities assumed.** Standard preprocessing, including JPEG |
| recompression, resize, crop, and mild blur. We do not assume a purification or |
| denoising defense, and we do not assume adversarially robust encoders. See Limitations. |
| - **Success criterion.** The supervision derived from the protected image is semantically |
| displaced from ground truth, for example when the assigned caption or label describes a |
| decoy class, so the image-label pair is misleading rather than clean. |
|
|
| ## 3. Method |
|
|
| Version 0.2.1 releases the ensemble PGD attack as a method-and-code repository. The |
| "model" is an optimization procedure over off-the-shelf public vision encoders, and no |
| weights are trained or distributed. |
|
|
| ### 3.1 Encoder ensemble |
|
|
| The perturbation is optimized against an attack set of 13 public vision encoders, and it |
| is evaluated on a separate held-out set of encoders that never take part in the |
| optimization. Keeping the held-out encoders out of the attack lets their flip rate |
| measure genuine cross-architecture transfer rather than self-similarity. |
|
|
| Attack set (13 encoders): |
|
|
| - **Contrastive CLIP-family** with text towers: OpenAI CLIP, LAION CLIP-H and CLIP-bigG, |
| EVA02, SigLIP and SigLIP2, MetaCLIP, and a ConvNeXt CNN for architectural variety. |
| - **Self-supervised and auto-supervised feature towers** with no text tower: AIMv2 and |
| DINOv2. |
| - **Modern VLM vision towers** with no text tower: MoonViT (Kimi), InternViT (InternVL), |
| and the MiniMax-M3 vision tower, integrated through hand-written differentiable loaders. |
|
|
| Held-out judges (never attacked): |
|
|
| - Two DFN CLIP encoders as contrastive anchors. |
| - **C-RADIOv3-H**, an agglomerative foundation model that distills CLIP, SigLIP2, DINOv2, |
| and SAM into one tower, so a perturbation that reaches it is moving directions shared |
| across architectures rather than CLIP idiosyncrasies. |
| - **The Qwen3-VL-8B vision tower**, a real frontier VLM captioner backbone and the closest |
| open proxy we have for the behavior of production captioners. |
|
|
| ### 3.2 Objective |
|
|
| Let $f$ be the L2-normalized embedding of the perturbed image and $f_\text{clean}$ |
| the embedding of the clean image. For a contrastive encoder, let $e_\text{truth}$ and |
| $e_\text{decoy}$ be the text embeddings of the true and decoy labels. The per-encoder |
| loss combines a targeted term that pulls the image toward the decoy and away from the |
| truth, and an untargeted repel term that pushes the image off its clean representation, |
| weighted by $w_t$ and $w_r$: |
| |
| $$ |
| \mathcal{L}_\text{contrastive} = |
| w_t \big[\cos(f, e_\text{decoy}) - \cos(f, e_\text{truth})\big] |
| - w_r \, \cos(f, f_\text{clean}). |
| $$ |
| |
| Feature-only towers have no text tower, so a text direction is not available. We instead |
| give each of them a decoy target computed from images. For a class $c$ and encoder |
| $g$, the centroid is the mean L2-normalized embedding of a set of exemplar images |
| $X_c$ of that class, |
|
|
| $$ |
| \mu_c^{(g)} = \frac{\displaystyle\sum_{x \in X_c} \hat{g}(x)} |
| {\big\lVert \sum_{x \in X_c} \hat{g}(x) \big\rVert_2}, |
| \qquad |
| \hat{g}(x) = \frac{g(x)}{\lVert g(x) \rVert_2}, |
| $$ |
| |
| and the feature-tower loss steers the image toward the decoy centroid and away from the |
| truth centroid: |
| |
| $$ |
| \mathcal{L}_\text{feature} = |
| w_t \big[\cos(f, \mu_\text{decoy}) - \cos(f, \mu_\text{truth})\big]. |
| $$ |
| |
| The exemplar images used to build the centroids are disjoint from the evaluation images, |
| which keeps class information from leaking into the benchmark. |
| |
| ### 3.3 EOT robustness |
| |
| Each optimization step averages gradients over Expectation-over-Transformation draws, |
| which include JPEG through a straight-through estimator, Gaussian blur, resize, and crop. |
| Averaging over these transforms is what lets the perturbation survive the recompression |
| and rescaling that destroy a naive PGD attack. |
| |
| ### 3.4 Transfer levers |
| |
| To avoid overfitting to any single encoder family, the attack normalizes each encoder's |
| gradient to unit L2 length before aggregation, samples a family-aware subset at every step |
| so no step is dominated by near-clone architectures, and can optionally use variance-tuned |
| momentum (VMI). |
| |
| ### 3.5 Perceptual budget |
| |
| An optional LPIPS penalty, combined with a hard LPIPS projection and a DCT low-pass mask, |
| trades transfer strength for lower visual salience. This yields a more strongly |
| imperceptible regime at a measurable cost in flip rate. |
| |
| ## 4. Experimental setup |
| |
| - **Data.** 60 Imagenette images, with a disjoint set of exemplar images used only to |
| build the feature-tower centroids. |
| - **Optimization.** 120 PGD steps, subset size 6, gradient normalization on, |
| `max_per_family = 2`, and `min_feature = 1`. |
| - **Post-processing.** Adversarial images are JPEG Q85 recompressed before evaluation. |
| - **Frontier judges.** GPT-5.5 and Gemini 3.5 Flash caption the recompressed image. We |
| count a flip only when the model labeled the clean image correctly, the label on the |
| protected image is a real answer rather than a refusal, that label is now far from the |
| truth, and it is not a synonym of the clean label. The rate is reported over the images |
| each model labeled correctly on the clean version. |
| - **Metrics.** Flip rate for efficacy, SSIM, LPIPS, PSNR, and ΔE2000 for stealth, and the |
| decoy-versus-truth margin on held-out encoders for open-model transfer. Confidence |
| intervals are 95% bootstrap intervals over images. |
| - **Hardware.** Developed on an NVIDIA RTX PRO 6000 (96 GB). A reduced contrastive-only |
| configuration fits smaller GPUs, and the full ensemble needs more memory. |
|
|
| ## 5. Results |
|
|
| Results are from a 60-image benchmark and should be read as a research signal rather than |
| a guarantee. Frontier flip rate is measured over the images each model labeled correctly |
| on the clean version, so the denominator differs by judge. |
|
|
| ### 5.1 Frontier captioners (flip rate over clean-correct trials) |
|
|
| | Configuration | GPT-5.5 | Gemini 3.5 Flash | Pooled | Pooled 95% CI | |
| |---|---|---|---|---| |
| | Max transfer (ε = 6, plain) | 37.1% | 52.6% | 45.2% | [33.8, 56.6] | |
| | Max stealth (ε = 6, perceptual) | 25.7% | 44.7% | 35.6% | [24.7, 46.8] | |
|
|
| ### 5.2 Open held-out transfer and stealth |
|
|
| | Configuration | Held-out transfer | Attack-set flip | SSIM | LPIPS | PSNR | ΔE2000 p95 | |
| |---|---|---|---|---|---|---| |
| | Max transfer (ε = 6, plain) | 28.5% [21.1, 36.1] | 67.3% [60.0, 74.1] | 0.939 | 0.134 | 33.8 dB | 8.1 | |
| | Max stealth (ε = 6, perceptual) | 17.1% [10.7, 24.1] | 54.0% [46.2, 61.5] | 0.960 | 0.023 | 35.7 dB | 6.8 | |
|
|
| **Findings.** Putting the full ensemble back into the attack and reserving fresh |
| cross-architecture encoders as judges gives the strongest frontier result we have measured, |
| and it holds up on encoders that never took part in the optimization. The perceptual regime |
| trades some of that transfer for higher stealth. We separately evaluated two very large |
| contrastive encoders (SigLIP2-giant and MetaCLIP2-H) as extra attackers, but a matched |
| ablation showed they slightly reduced both frontier and held-out transfer, so they are not |
| part of the released attack. Adding more CLIP-family capacity is not the bottleneck; |
| architectural diversity is. |
|
|
| Clean image (left) versus protected image (right): |
|
|
|  |
|
|
| A full before/after grid is provided in |
| [`assets/grid_before_after_16.png`](assets/grid_before_after_16.png). |
|
|
| ## 6. Limitations |
|
|
| - **Not a permanent opt-out.** Aggressive downscaling, denoising, adversarial |
| purification, heavy recompression, or future adversarially robust training pipelines can |
| reduce or remove the perturbation. |
| - **Transfer is imperfect.** The perturbation is optimized on public encoders and only |
| partially transfers to closed or evolving production systems. |
| - **Pipeline-dependent.** The measured effect is strongest against pipelines that rely on |
| captioners, taggers, CLIP-like encoders, or VLM synthetic labels. |
| - **Stealth and strength trade off.** More imperceptible settings measurably reduce |
| transfer. |
| - **Still an early benchmark.** Headline numbers come from 60 images, and broader |
| evaluation across datasets and pipelines is future work. |
|
|
| ## 7. Ethics and intended use |
|
|
| VEIL-PGD is intended for defensive use, helping people make their own images less reliable |
| as training data when consent and opt-out mechanisms are absent, opaque, or unenforced. It |
| should not be used to degrade datasets you do not own, to evade content-moderation or |
| safety systems, or to mislead people about the visible content of an image. |
|
|
| ## 8. Future work |
|
|
| - Broaden the held-out judges further so transfer is measured against as many distinct |
| architectures as possible. |
| - Add an architecturally distinct attacker rather than another CLIP-family encoder, since |
| the ablation above shows extra CLIP capacity does not help. |
| - Scale the benchmark across more datasets and more realistic ingestion pipelines. |
| - Push the perceptual regime to recover transfer at high stealth. |
|
|
| ## 9. Reproducibility |
|
|
| The attack runs on a CUDA GPU. |
|
|
| ```bash |
| python3.11 -m venv .venv |
| . .venv/bin/activate |
| pip install -r requirements-lock.txt |
| |
| # CSV manifest: "image_path,truth_label" per line |
| python -m ensemble.run_attack \ |
| --manifest examples/testset60.csv --images examples/testset60 \ |
| --exemplars examples/exemplars --train v0.2.1 \ |
| --out runs/demo --steps 120 --eps 6 --subset 6 \ |
| --grad-norm --max-per-family 2 --min-feature 1 --metrics |
| ``` |
|
|
| Key flags: `--eps` sets the L-infinity budget in `/255`, `--train` selects the encoder |
| set (`v0.2.1` is the released attack plus the cross-architecture judges), `--exemplars` |
| points at the per-class images used for feature-tower centroids, `--lpips-weight`, |
| `--lpips-tau`, and `--dct-keep` control the perceptual budget, and `--metrics` records |
| PSNR, SSIM, ΔE, and LPIPS. Adversarial PNGs are written to `runs/<out>/adv/`, and |
| `results.json` records per-encoder margins together with attack-set and held-out flip |
| rates. |
|
|
| ### Repository layout |
|
|
| ```text |
| ensemble/ the attack implementation |
| attack.py EOT, ensemble, and momentum-PGD loop |
| encoders.py differentiable encoder interface and the train/judge split |
| loss.py per-encoder gradient aggregation and VMI support |
| sampling.py family-aware subset sampling |
| perceptual.py LPIPS, DCT low-pass, and stealth metrics |
| targets.py feature-tower decoy centroids from exemplar images |
| eot.py JPEG-STE, blur, resize, and crop transforms |
| run_attack.py attack runner and margin scoring |
| towers/ differentiable loaders for VLM and foundation vision towers |
| src/veil_pgd/ legacy typographic-overlay CLI and service adapters |
| scripts/ frontier evaluation, aggregation, and test-set builders |
| research/ design notes, scouting reports, and results |
| services/ GPU inference service for legacy experiments |
| ``` |
|
|
| ## 10. Citation |
|
|
| ```bibtex |
| @software{veil_pgd_2026, |
| author = {cebeuq}, |
| title = {VEIL-PGD: Vision-Encoder Imperceptible Layer for Training-Data Protection}, |
| year = {2026}, |
| version = {0.2.1}, |
| url = {https://huggingface.co/cebeuq/veil-pgd}, |
| note = {EOT-hardened ensemble-PGD image-side protection against |
| vision-language training-data ingestion.} |
| } |
| ``` |
|
|
| ## 11. License |
|
|
| **PolyForm Noncommercial License 1.0.0**, see `LICENSE`. Free for noncommercial personal, |
| research, educational, nonprofit, and government use. Commercial use is not granted under |
| this license, so please contact the authors for a commercial arrangement. |
|
|
