title: >-
Where Lesions Live: Mid-Layer Localization in Frozen Vision Transformers, and
Why
status: working draft (paper
venue_targets:
- SPIE Medical Imaging
- MIDL
- workshop on representation analysis
Where Lesions Live: Mid-Layer Localization in Frozen Vision Transformers, and Why
Abstract
We show that the lesion-localizable signal in frozen self-supervised vision transformers lives in mid/early layers, not the final layer, that the optimal layer is selectable without labels, and that the decline with depth is caused by representation globalization (view-invariance), not loss of spatial information — a mechanism that holds across self-distillation and supervised objectives but is absent for masked reconstruction (which never localizes). A label-free density/membership probe over patch tokens localizes lesions; its AUROC on chest CT rises from 0.565 at the final block to 0.871 at block 3 (peak 0.866 ± 0.010 over 3 seeds), and the tail-gap of the membership-score distribution selects a layer within 0.006 AUROC of the masked oracle (multi-seed) — entirely label-free. Across three objectives — DINOv2 (self-distillation, peak 0.88), supervised ViT (peak 0.84), MedDINOv3 (CT self-distillation) — localizability peaks early/mid and erodes with depth, strongly anti-correlated with rising flip-invariance (ρ = −0.73 to −0.94). Masked-reconstruction (MAE) features are not density-separable for lesions at any depth (≈0.59 flat). Implication: for dense localization in frozen ViTs, read the mid layer of a self-distillation/supervised backbone, found label-free.
1. The finding (depth)
Token-level lesion-membership AUROC by block (LIDC, MedDINOv3): final 0.565 → block 6 0.769 → block 4 0.865 → block 3 0.871 (Fig. 1; multi-seed peak 0.866 ± 0.010, monotone decline to 0.637 ± 0.026). Final-layer features serve the global self-distillation objective; the dense local lesion signal is mid/early.
2. Label-free layer selection
We select the operating layer with NO masks from the shape of the membership-score distribution.
The tail-gap (q99−q50)/(q50−q01) is the robust selector: multi-seed regret 0.006 AUROC
(max 0.011) versus the mask-derived oracle. A bimodality statistic correlates with the depth curve
(ρ=0.69) but is less stable across seeds (regret 0.062), and excess kurtosis is a poor proxy (picks
the worst layer). So where to read is discoverable from the score distribution alone, without
annotation.
3. The mechanism (why mid-layer)
We disentangle two candidate causes per layer: spatial information (position-probe accuracy) and globalization (flip-invariance), measured on MedDINOv3. Localizability anti-correlates with flip-invariance (ρ=−0.94), not with spatial information — patch position is near-perfectly decodable at every block, so the loss with depth is not positional. This is architecturally expected for this backbone: MedDINOv3/DINOv3 encode position with axial RoPE applied to the queries/keys of every attention block (patch tokens only; CLS and register/storage tokens excluded), with no learned absolute position embedding — so positional information is re-injected at all depths by construction (verified against the DINOv3 source). DINOv2, our ultrasound backbone, differs: it uses a learned absolute position embedding added once at the input (not RoPE); there we confirm the invariance–localizability coupling empirically (§4, ρ=−0.93) but do not import the RoPE-based positional control. As features become invariant to augmentation (the self-distillation goal), they trade away the fine local discrimination small lesions need.
4. Cross-objective: the mechanism is causal-by-comparison
Holding training domain constant (natural-image backbones, evaluated on CT):
| objective | peak AUROC | final | ρ(invariance, AUROC) |
|---|---|---|---|
| DINOv2 (self-distillation) | 0.880 (blk2) | 0.617 | −0.93 |
| ViT (supervised) | 0.842 (blk1) | 0.658 | −0.73 |
| MAE (reconstruction) | 0.611 (blk1) | 0.568 | +0.06 |
Depth-erosion + invariance-coupling hold for both objectives that produce a localizer (self-distillation, supervised), and for CT-native MedDINOv3 (ρ=−0.94) — three objectives, same law (Fig. 6). MAE is the clarifier: it is flat and low (0.59) — masked reconstruction features are not density-separable for lesions at any depth, so "no collapse" is trivial (nothing to lose). The method needs self-distillation/supervised features; reconstruction is the wrong pretext.
5. Rigor
Multi-seed (n=3): the depth curve peaks at block 3 = 0.866 ± 0.010 and declines monotonically
to 0.637 ± 0.026 (std grows with depth). The label-free tail-gap selector regret is 0.006 (max
0.011) across seeds — robust; bimodality is less reliable multi-seed (0.062), so tail-gap is the
headline selector. Cross-objective curves span all 12 blocks per backbone.
[research_v3/rigor_results.json, research_v3/f3_cross_objective.json.]
6. Implications & reconciliation with the probe literature
- Read the mid layer of a self-distillation/supervised ViT for dense localization; find it label-free via membership-distribution bimodality.
- A representation-coverage probe evaluated on final-layer features is reading the wrong layer; this work gives the corrected depth and the mechanism. (Reconcile / cross-cite the companion probe study to reinforce rather than overlap.)
Appendix — artifacts
research_v2/s2_depth_localizability.json, research_v3/f2b_f3a_results.json,
research_v3/f3_cross_objective.json, research_v3/rigor_results.json. Figures: paper/figures/ fig1_layer_ablation.png (depth, error bars), fig6_cross_objective.png (mechanism). HF-Job reproducible.