---
title: "Where Lesions Live: Mid-Layer Localization in Frozen Vision Transformers, and Why"
status: working draft (paper #3 — mechanism / SPIE probe)
venue_targets: [SPIE Medical Imaging, MIDL, workshop on representation analysis]
---

# Where Lesions Live: Mid-Layer Localization in Frozen Vision Transformers, and Why

## Abstract

We show that the lesion-localizable signal in frozen self-supervised vision transformers lives in
**mid/early layers, not the final layer**, that the optimal layer is **selectable without labels**,
and that the decline with depth is **caused by representation globalization (view-invariance), not
loss of spatial information** — a mechanism that holds across self-distillation and supervised
objectives but is absent for masked reconstruction (which never localizes). A label-free
density/membership probe over patch tokens localizes lesions; its AUROC on chest CT rises from
0.565 at the final block to 0.871 at block 3 (peak 0.866 ± 0.010 over 3 seeds), and the **tail-gap**
of the membership-score distribution selects a layer within **0.006 AUROC** of the masked oracle
(multi-seed) — entirely label-free.
Across three objectives — DINOv2 (self-distillation, peak 0.88), supervised ViT (peak 0.84),
MedDINOv3 (CT self-distillation) — localizability peaks early/mid and erodes with depth, strongly
anti-correlated with rising flip-invariance (ρ = −0.73 to −0.94). Masked-reconstruction (MAE)
features are not density-separable for lesions at any depth (≈0.59 flat). Implication: for dense
localization in frozen ViTs, read the mid layer of a self-distillation/supervised backbone, found
label-free.

## 1. The finding (depth)

Token-level lesion-membership AUROC by block (LIDC, MedDINOv3): final 0.565 → block 6 0.769 →
block 4 0.865 → **block 3 0.871** (Fig. 1; multi-seed peak 0.866 ± 0.010, monotone decline to
0.637 ± 0.026). Final-layer features serve the global self-distillation objective; the dense local
lesion signal is mid/early.

## 2. Label-free layer selection

We select the operating layer with NO masks from the shape of the membership-score distribution.
The **tail-gap** `(q99−q50)/(q50−q01)` is the robust selector: multi-seed regret **0.006 AUROC**
(max 0.011) versus the mask-derived oracle. A bimodality statistic correlates with the depth curve
(ρ=0.69) but is less stable across seeds (regret 0.062), and excess kurtosis is a poor proxy (picks
the worst layer). So *where to read* is discoverable from the score distribution alone, without
annotation.

## 3. The mechanism (why mid-layer)

We disentangle two candidate causes per layer: spatial information (position-probe accuracy) and
globalization (flip-invariance), measured on MedDINOv3. **Localizability anti-correlates with
flip-invariance (ρ=−0.94), not with spatial information** — patch position is near-perfectly
decodable at *every* block, so the loss with depth is not positional. This is architecturally
expected for this backbone: MedDINOv3/DINOv3 encode position with **axial RoPE applied to the
queries/keys of every attention block** (patch tokens only; CLS and register/storage tokens
excluded), with *no* learned absolute position embedding — so positional information is re-injected
at all depths by construction (verified against the DINOv3 source). DINOv2, our ultrasound backbone,
differs: it uses a **learned absolute position embedding added once at the input** (not RoPE); there
we confirm the invariance–localizability coupling empirically (§4, ρ=−0.93) but do not import the
RoPE-based positional control. As features become invariant to augmentation (the self-distillation
goal), they trade away the fine local discrimination small lesions need.

## 4. Cross-objective: the mechanism is causal-by-comparison

Holding training domain constant (natural-image backbones, evaluated on CT):

| objective | peak AUROC | final | ρ(invariance, AUROC) |
|---|---|---|---|
| DINOv2 (self-distillation) | 0.880 (blk2) | 0.617 | −0.93 |
| ViT (supervised) | 0.842 (blk1) | 0.658 | −0.73 |
| MAE (reconstruction) | 0.611 (blk1) | 0.568 | +0.06 |

Depth-erosion + invariance-coupling hold for both objectives that produce a localizer
(self-distillation, supervised), and for CT-native MedDINOv3 (ρ=−0.94) — three objectives, same
law (Fig. 6). **MAE is the clarifier:** it is flat *and low* (0.59) — masked reconstruction features are not
density-separable for lesions at any depth, so "no collapse" is trivial (nothing to lose). The
method needs self-distillation/supervised features; reconstruction is the wrong pretext.

## 5. Rigor
Multi-seed (n=3): the depth curve peaks at **block 3 = 0.866 ± 0.010** and declines monotonically
to 0.637 ± 0.026 (std grows with depth). The label-free **tail-gap selector regret is 0.006 (max
0.011)** across seeds — robust; bimodality is less reliable multi-seed (0.062), so tail-gap is the
headline selector. Cross-objective curves span all 12 blocks per backbone.
[`research_v3/rigor_results.json`, `research_v3/f3_cross_objective.json`.]

## 6. Implications & reconciliation with the probe literature
- Read the **mid layer** of a self-distillation/supervised ViT for dense localization; find it
  **label-free** via membership-distribution bimodality.
- A representation-coverage probe evaluated on **final-layer** features is reading the wrong layer;
  this work gives the corrected depth and the mechanism. (Reconcile / cross-cite the companion
  probe study to reinforce rather than overlap.)

### Appendix — artifacts
`research_v2/s2_depth_localizability.json`, `research_v3/f2b_f3a_results.json`,
`research_v3/f3_cross_objective.json`, `research_v3/rigor_results.json`. Figures: `paper/figures/
fig1_layer_ablation.png` (depth, error bars), `fig6_cross_objective.png` (mechanism). HF-Job reproducible.