# Agent 2 Report: OSS-to-Closed Transfer and Ensemble-over-Encoders Training

## 1. TL;DR verdict

Transfer is the weak link. The evidence supports transfer for sufficiently strong semantic typography and for carefully engineered pixel/feature attacks trained on encoder ensembles, but it does not support assuming that strict-imperceptible one-word overlays found on small OSS VLMs will transfer to GPT-class or Gemini-class captioners.

The most defensible pivot is not "replace Tier A with ViT displacement." It is: add an ensemble encoder feature objective, explicitly search a visible-strength ladder, and spend black-box queries measuring the crossover. If the product must remain at the current strict stealth gate, expect low frontier success.

No public measurement found for the exact current targets `openai/gpt-5.5` or `google/gemini-3.5-flash`; numbers below are extrapolated from GPT-4o/Gemini 1.5/2.0/Claude/Qwen/LLaVA literature plus our first project sweep. I flag those extrapolations as speculative.

## 2. Findings with citations

### 2.1 What is actually known about transfer

**Exact target gap.** I found no public adversarial or typographic transfer study measuring `gpt-5.5` or `gemini-3.5-flash` directly. The relevant public proxies are GPT-4V/GPT-4o/GPT-4.1, Gemini 1.5 Pro/Gemini 2.0, Claude 3.x/4.x, Qwen-VL/Qwen2.5/3-VL, LLaVA, InternVL, CLIP/SigLIP-family encoders.

**Pixel/feature attacks can transfer to closed models when engineered for transfer.** This is stronger evidence than I expected, but it does not directly validate faint text overlays.

| Source | Surrogate/training setup | Closed/proxy target results | Why it matters |
|---|---:|---:|---|
| Dong et al., "How Robust is Google's Bard..." / Attack-Bard | White-box surrogate vision encoders or MLLMs | Bard image-description ASR 22%; GPT-4V 45%; Bing Chat 26%; ERNIE Bot 86% | Early evidence that surrogate-crafted visual perturbations transfer to proprietary MLLMs, but rates were inconsistent. URL: https://github.com/thu-ml/Attack-Bard |
| Hu et al. 2025, "Transferable Adversarial Attacks on Black-Box Vision-Language Models" | 8 CLIP-family surrogate encoders, targeted image-captioning perturbations | At eps=8/255: GPT-4o 83.9%, Gemini 1.5 Pro 70.2%, Claude 3.5 15.1%, Claude 3.7 21.3%. At eps=16/255: GPT-4o 94.4%, Gemini 1.5 Pro 86.1%, Claude 3.5 58.7%, Claude 3.7 62.7%. | Strongest direct evidence that many CLIP/SigLIP-family encoders improve closed-model transfer. Not typography; uses norm-bounded pixel perturbations and stronger compute than our current stack. URL: https://arxiv.org/abs/2505.01050 |
| Jia et al. 2025, FOA-Attack | CLIP ViT-B/16, ViT-B/32, ViT-g-14-laion ensemble; global CLS alignment + local patch-token optimal-transport alignment + dynamic ensemble weights | Closed MLLM targeted caption ASR: GPT-4o 75.1%, GPT-4.1 77.3%, Gemini 2.0 53.4%, Claude 3.5 11.9%, Claude 3.7 15.8%; reasoning targets: GPT-o3 81%, Gemini-2.0-flash-thinking 57%, Claude-thinking 16%. | Shows patch-token/local feature alignment and dynamic weights help, but Claude remains a hard transfer target. Uses A100-scale setup and pixel perturbations. URL: https://arxiv.org/html/2505.21494 |
| Wu et al. 2024/ICLR 2025, multimodal agents | CLIP/captioner attacks against web agents | GPT-4V agent: CLIP attack 21% without captioner, 43% when GPT-4V self-captions; captioner-augmented attack up to 75%; abstract reports up to 67% in a broader agent setting. | Agent setting, not image captioning, but reinforces that transfer depends heavily on the downstream perception pipeline. URL: https://arxiv.org/abs/2406.12814 |

**Typographic attacks transfer semantically, but the rates depend wildly on model, prompt, target word, and text visibility.**

OpenAI's original CLIP interpretability work already identified "typographic attacks": CLIP neurons respond to text in images, and handwritten text can fool the model in the wild. This is the root mechanism we are trying to exploit, but it was qualitative, not a closed-frontier transfer benchmark. URL: https://openai.com/index/multimodal-neurons/

TVPI (2025) gives the most useful modern cross-model typographic table. It evaluates open and closed LVLMs with rendered visual prompts. Key VLP ASR values from Table 3:

| Model | Relevant TVPI ASR examples |
|---|---:|
| Qwen-v2.5-VL-7B | Near-zero: e.g. "sorry" 0.003, "unknown" 0.002 |
| Qwen-v2.5-VL-72B | Very high: "sorry" 0.903, "unknown" 0.917, "suicide" 0.795, "Asian" 0.870 |
| LLaVA-v1.6-7B/13B/34B | Near-zero in their setup |
| LLaVA-v1.6-72B | High: "sorry" 0.688, "murder" 0.769 |
| InternVL-v2.5-38B | Moderate: "sorry" 0.263, many targets lower |
| InternVL-v2.5-78B | Low: mostly 0.011-0.054 |
| GPT-4o | High on protective targets: "sorry" 0.600, "unknown" 0.765; much lower on harmful/bias targets: 0.005-0.190 |
| Claude-3.5-Sonnet | "sorry" 0.665, "unknown" 0.580, "murder" 0.480, "African" 0.645 |

Source: https://arxiv.org/html/2503.11519v4

Important implication for us: small Qwen/LLaVA surrogates are not reliable indicators of closed frontier behavior. In TVPI, Qwen-7B was robust while Qwen-72B and GPT-4o were vulnerable; LLaVA small variants were robust while LLaVA-72B was vulnerable. Our current `qwen-3.5-4b` and `gemma-4-4b` being fooled at strict stealth says little about GPT/Gemini.

**Single-word decoys are weaker than instruction-like visual prompts.** TVPI explicitly reports that reducing the visual prompt to only the attack target word has much less impact than full typographic visual prompts. That directly cuts against our current one-word/short-label overlay strategy if the goal is to flip a strong closed captioner far from truth. URL: https://arxiv.org/html/2503.11519v4

**Real-world typographic data supports the vulnerability, not our exact stealth setting.** SCAM/SynthSCAM reports 1,162 real-world typographic attack images, average VLM accuracy drop of 26%, synthetic drop of 35%, and 30-50% drops in smaller LLaVA variants; it also finds larger LLM backbones can mitigate vision-encoder weakness. URL: https://bliss-e-v.github.io/SCAM-project-page/

**CLIP text-image matching transfers at least to another open LVLM.** "Typographic Attacks in a Multi-Image Setting" reports that choosing attack text by CLIP text-image similarity improves CLIP ASR by 21% over random and that this CLIP-derived text-image similarity generalizes when attacking InstructBLIP. This supports using embedding/feature proxies, but it does not establish transfer to GPT/Gemini. URL: https://arxiv.org/html/2502.08193v1

### 2.2 Imperceptible vs legible: where is the crossover?

The crossover is not a single universal threshold. The most concrete pattern is: below OCR/readability threshold, GPT/Claude-class models often do not follow the visual text; once text is reliably readable, semantic transfer improves, but safety/alignment and task framing still matter.

**Font size threshold.** A 2026 controlled study rendered 1,000 SALAD-Bench prompts as typographic images at 6-28 px and tested GPT-4o, Claude Sonnet 4.5, Mistral-Large-3, and Qwen3-VL-4B. It found near-zero or low ASR at 6 px and a plateau beginning around 10-12 px. Exact image-ASR by font size:

| Font | GPT-4o | Claude | Mistral | Qwen3-VL |
|---:|---:|---:|---:|---:|
| 6 px | 0.3 | 1.2 | 15.0 | 23.9 |
| 8 px | 3.5 | 10.5 | 67.9 | 40.8 |
| 10 px | 6.4 | 21.6 | 73.5 | 43.1 |
| 12 px | 7.0 | 18.8 | 75.5 | 42.7 |
| 20 px | 7.7 | 16.4 | 73.8 | 48.2 |
| Text baseline | 35.6 | 46.6 | 85.0 | 48.9 |

Source: https://arxiv.org/html/2604.12371

That study is about harmful prompt compliance, not training data protectioning, and it uses black text on white backgrounds, not faint blended overlays. Still, its threshold finding is highly relevant: for GPT-4o and Claude, simply making text legible does not guarantee high image-channel ASR.

**Opacity/contrast threshold.** "Unveiling Typographic Deceptions" reports that LLaVA-v1.5 object-recognition accuracy still drops at 6 px and 20% opacity, and that visibility correlates with attack strength. In their appendix table for object recognition, LLaVA-v1.5 accuracy under typography was 81.2% at 3 px, 61.0% at 6 px, 47.6% at 9 px, 42.6% at 12 px, 40.0% at 15 px; for opacity it was 97.6% at 20%, 86.8% at 40%, 50.8% at 60%, 40.2% at 80%, 39.6% at 100%. Source: https://arxiv.org/html/2402.19150v3

This shows a faint band exists for some open models, but it is not closed-frontier evidence.

**Stealth text in natural-image regions has one useful black-box data point.** The 2026 IPI paper uses segmentation, adaptive font scaling, and background-aware rendering. It reports:

- Pixel-level blending: visually strongest stealth, max ASR only 10%.
- Background patch coloring: single-mask peak 19% at +20 brightness; multi-mask range 15-25% around +15 to +25.
- Global region-averaged coloring: base prompt at +20 reached 41%; object-aware prefix plus base prompt reached 64% at +20 and 52% at +15; offset 0 was 18%.

Source: https://arxiv.org/html/2603.03637v1

This is the best evidence for a "faint but semantically legible" band, but it is only GPT-4-turbo in that paper, it is prompt-injection rather than caption-label displacement, and the stealth metrics are not identical to our PSNR/SSIM/LPIPS/dE gate. Treat the 52-64% numbers as an upper-bound signal, not a prediction for our current one-word decoys.

**Imperceptible steganographic/pixel instruction attacks are weaker.** "Invisible Injections" reports 24.3% +/- 3.2% overall ASR across GPT-4V, Claude, and LLaVA, neural steganography up to 31.8%, while maintaining PSNR > 38 dB and SSIM > 0.94. Source: https://arxiv.org/abs/2507.22304

My read: strict imperceptibility can work sometimes, but not reliably enough for this product unless the product accepts a low success rate or pays for target-specific black-box tuning.

### 2.3 Does training on an encoder ensemble improve transfer?

Yes, measurably, for pixel/feature attacks. For typography specifically, the evidence is indirect but still actionable because the same OCR/semantic feature alignment is the bottleneck.

Evidence:

1. Hu et al. 2025 show a clear surrogate-count effect. With 3 CLIP encoders, closed-target image-caption ASR at eps=16/255 was GPT-4o 85.1%, Claude 3.5 36.3%. With 8 CLIP encoders, it rose to GPT-4o 94.4%, Claude 3.5 58.7%. Adding Qwen2.5-VL, DINOv2, or adversarially trained models to only 3 CLIPs helped little; adding non-CLIP models to 8 CLIPs did not improve much. URL: https://arxiv.org/abs/2505.01050

2. FOA-Attack improves over global-feature matching by aligning local patch tokens and dynamically weighting encoders. It reports GPT-4o 75.1% and Gemini-2.0 53.4% closed ASR, versus M-Attack 60.3% and 44.8% respectively, under the same comparison table. URL: https://arxiv.org/html/2505.21494

3. X-Transfer scales universal CLIP attacks over 16/32/64 encoder search spaces and dynamically selects a subset per step. It reports that increasing surrogate search-space size improves transfer across CLIP encoders and downstream VLM tasks, and it evaluates VLM captioning/VQA on OpenFlamingo, LLaVA, MiniGPT-4, and BLIP2. URL: https://arxiv.org/html/2505.05528v3

4. General adversarial-transfer literature supports ensemble attacks but warns that naive averaging is not optimal. "Rethinking Model Ensemble in Transfer-based Adversarial Attacks" frames transfer as finding common weaknesses with flat loss landscape and proximity to each model optimum; it also applies ensemble transfer to Google's Bard. URL: https://arxiv.org/abs/2303.09105

Concrete recipe for our purposes:

- Start with 3 encoders on klaus-3, loaded sequentially if VRAM is tight:
  - `open_clip ViT-B-32/laion2b_s34b_b79k` (already in `vision_service.py`).
  - `open_clip ViT-L-14/laion2b_s32b_b82k` or `ViT-L-14/openai`.
  - A SigLIP-family encoder, preferably `ViT-B/16-SigLIP` if using `transformers`, or `ViT-SO400M/14-SigLIP` only if model-manager frees enough VRAM.
- If we use cloud or temporarily stop the resident llama.cpp VLMs, expand to 6-8 CLIP/SigLIP-family encoders:
  - CLIP ViT-B/32, ViT-B/16, ViT-L/14, ViT-L/14@336
  - OpenCLIP ViT-H/14 or ViT-g/14 LAION/DFN if memory allows
  - SigLIP B/16 and SigLIP SO400M/14
  - Optional ConvNeXt-L/XXL CLIP variant
- Do not prioritize full Qwen/InternVL/LLaVA decoders in the first encoder ensemble. The literature suggests many CLIP-family encoders give most of the transfer benefit, while full VLLMs are expensive and do not clearly improve once the CLIP ensemble is broad enough.
- Score candidates under expectation over transforms: original, JPEG Q85, light blur, resize/downsample. For each encoder `i` and transform `t`, compute:
  - `d_clean_i = 1 - cos(f_i(T(render(x, spec))), f_i(T(x)))`
  - `gap_decoy_i = cos(f_i(T(render(x, spec))), text_i(decoy)) - cos(f_i(T(render(x, spec))), text_i(truth))`
  - optional OCR/legibility proxy: `cos(f_i(crop_text_region), text_i(rendered_text))`
- Candidate score:
  - `score_i = z(d_clean_i) + 0.5*z(gap_decoy_i) + 0.25*z(legibility_i)`
  - aggregate with a soft-min or dynamic weighting, not a plain mean: `score = softmin_i(score_i / tau)` or upweight encoders whose scores lag. This avoids a candidate that only fools one encoder.
- Use the encoder score to rank a strength ladder, not to make the final decision. Final selection still needs sparse GPT/Gemini Tier B validation.

For differentiable training, restrict continuous optimization to a fixed glyph mask: choose text/position/font from discrete search, then optimize alpha/color/brightness over the encoder ensemble. For the current non-differentiable renderer, CMA-ES/TPE/ES over `RenderSpec` is more practical than backprop end-to-end.

## 3. Concrete recommendation for our stack

Do not pivot to "ViT displacement only." Add an encoder-ensemble objective as an additional Tier A scorer and force exploration across attack strength. The present strict/medium/loose sweep failed because the surrogate decoder label was saturated at strict; a feature objective plus explicit strength ladder fixes that measurement bug.

Implementation shape:

1. Extend `services/vision_service.py` on klaus-3 with encoder feature scoring.
   - Keep the Mac torch-free.
   - Add endpoints like `/encoders/list`, `/encoders/features`, and `/encoders/score_overlay` or a batched `/encoders/score`.
   - Return normalized global embeddings first; add patch-token/local-feature support only after the global version works.
2. Add a `veil_pgd.targets.encoder` client and a feature scorer in `optimizer/`.
   - Minimal integration: `TierA` calls a new `FeatureScorer` after rendering candidates and adds `feature_score` to `Candidate.notes`.
   - Better integration: a separate `TierAFeature` ranks candidates by feature score, then decoder surrogates and Tier B validate.
3. Make strength an explicit dimension in `search_space.py`.
   - Font px: 8, 10, 12, 16, 20, 24 at 1024 reference.
   - Alpha: 0.15, 0.25, 0.35, 0.50.
   - Brightness offset: 0, 10, 15, 20, 30.
   - Repetition: 1-3.
   - Include "this is [decoy]" / "[decoy]" / "a photo of [decoy]" variants in the decoy text pool and measure which is acceptable for product aesthetics.
4. For 8 GB VRAM, run encoders sequentially or temporarily pause llama.cpp VLMs during feature-search batches.
   - Current resident stack is already around 6.1/8 GB. A 3-encoder ensemble will not comfortably coexist if all encoders are loaded at once.
   - Sequential scoring is acceptable because encoder inference is far cheaper than decoder VLM calls.
5. Black-box validation must sample by strength level.
   - Send top candidates from each strength bucket to GPT/Gemini, not just top overall surrogate candidates.
   - Otherwise the optimizer will again select faint candidates that saturate local surrogates and teach us nothing about the frontier crossover.

## 4. Risks and uncertainty

**Speculative target extrapolation.** No public results for `gpt-5.5` or `gemini-3.5-flash` were found. All expected rates are extrapolated from earlier closed models plus our local n=3 result.

**Task mismatch.** Much of the high-ASR typography literature is jailbreak/prompt-injection. Our goal is untargeted semantic label displacement during captioning. A model can read "blush" or "murder" in the image and still answer "golf ball" when asked for the main object.

**One-word overlay weakness.** Evidence says full visual prompts outperform target-word-only typography. If the product only allows a subtle decoy word, success should be discounted.

**Encoder displacement may not predict decoder captions.** It is a useful proxy, not a guarantee. Decoder instruction hierarchy, OCR preprocessing, captioner prompts, and safety filters can erase the encoder effect.

**Closed models may use stronger OCR and better ignore-image-text policies.** GPT-class and Gemini-class systems can read faint text, but they may also distinguish "text present in image" from "main object in image." That is exactly what our current small surrogates fail to do.

**Compute.** The strongest transfer papers use 8+ encoders, A100-class hardware, or many optimization steps. Our 8 GB 3070 Laptop can run a pragmatic 3-encoder sequential scorer, but not the full research setup with VLMs resident.

**Defenses.** OCR stripping, prompt prefixes that tell models to ignore image text, aggressive resize/blur, and safety-tuned caption prompts all reduce success. The product remains an arms-race mitigation, not durable protection.

## 5. Realistic black-box expectations

These are speculative ranges for a 30-50 image standard set, success defined like our project: semantic distance from truth >= tau, stealth gate passed for that level, and JPEG Q85 survival. Error bars are broad because the exact targets are unmeasured.

| Strength | Plausible GPT-5.5 success | Plausible Gemini-3.5-flash success | Fool-both/min aggregate | Basis |
|---|---:|---:|---:|---|
| Strict imperceptible | 0-10%, center ~5% | 5-20%, center ~10% | 0-5% | Our strict sweep effectively got 0 true frontier flips on iconic/ordinary images; 6-8 px GPT/Claude typography ASR is low; IPI pixel blending maxed at 10%; stego prompt injection reports 24.3% +/- 3.2% but for a different task. |
| Mildly visible, semantically legible | 15-35%, center ~25% | 25-50%, center ~35% | 10-25% | IPI reports 41-64% at +15/+20 style offsets for instruction-like prompts; SCAM average VLM drop is 26%; TVPI closed GPT/Claude can reach 60%+ on some prompt targets but is target-dependent; simple rendered image ASR on GPT/Claude can remain low. |
| Mildly visible plus instruction-like wording ("this is [decoy]") | 25-50%, center ~35% | 35-60%, center ~45% | 15-35% | Upper-bound extrapolation from TVPI/IPI. This may be too visible or aesthetically unacceptable for the product. |

For the current implementation without objective changes, I would expect strict success to remain near zero on frontier models and mild/loose success to remain under-measured because the optimizer does not explore loud enough candidates.

## 6. First measurable milestone to ship

Ship a "feature-transfer pilot" before rebuilding the whole optimizer:

1. Add a 3-encoder sequential feature scorer to klaus-3: CLIP ViT-B/32, CLIP ViT-L/14, one SigLIP-family encoder.
2. Run the existing 40-image Imagenette test set with a forced strength ladder:
   - strict, medium, loose gates
   - font sizes 8/10/12/16/20
   - alpha 0.15/0.25/0.35/0.50
   - brightness offsets 10/15/20/30
3. For each image and strength level, black-box validate top 3 candidates by feature score and top 1 by current decoder-surrogate score against both GPT-5.5 and Gemini-3.5-flash.
4. Report per-target and fool-both success@0.3/0.5/0.7, not just max distance across all models.

Decision rule: if the encoder ensemble cannot produce at least ~20% black-box success@0.5 at medium/loose on the 40-image set, the white-box encoder pivot is not carrying the product. At that point, either accept visible instruction-like overlays, put frontier models directly into the optimization loop, or reject the pivot.

Repo note: I found no `*.plan.md` file in the repo root during this pass.