RESEARCH_OVERVIEW.md

covtoken — Research Overview

Label-free rare-structure geometry in frozen self-supervised vision transformers, for medical imaging. One principle, three papers, four gated programs — all compute run as Hugging Face Jobs, all experiments reproducible, labels eval-only throughout.

The principle (one sentence)

Rare/critical structure lives in the mid-layer concentration subspace of self-distilled foundation models; depth (view-invariance/globalization) erodes it; rank/spanning objectives are anti-aligned with it; and it is readable, certifiable, and bounded label-free.

The three papers

1. Method — Where Lesions Live: Label-Free Mid-Layer Lesion Subspaces for Token-Economical Medical Imaging (paper/working_draft.md). The label-free lesion subspace + membership pruning
   • conformal retention certificate + lesion-routed depth, across CT (MedDINOv3) and ultrasound (DINOv2). Includes the coverage-floor negative result.
2. Theory / negative-results — Rank-Based Representation Objectives Fail for Rare-Signal Retention: A Mechanism and a Predictive Law (paper/paper2_rank_objectives_draft.md). The exact closed-form law (gap=(m−r)/m, r*=m), the A(rank,SNR) alignment surface, the selection-vs-scaling sharpening. Transferable beyond medicine.
3. Mechanism / SPIE — Where Lesions Live: Mid-Layer Localization in Frozen ViTs, and Why (paper/paper3_midlayer_draft.md). Mid-layer localization + label-free tail-gap selector + the invariance mechanism + the cross-objective causal-by-comparison result.

The four programs (gated, HF-Jobs reproducible)

• v1 (method) — gate_reports/ Gates 0–6. Subspace validity, cross-modality localization, membership > saliency pruning, coverage-floor NEGATIVE, conformal cert (0.978 ≥ 0.90), routed depth (1.6× FLOPs @ 98% sensitivity).
• v2 (S1–S5) — research_v2/. S1 the closed-form law (✅), S2 mid-layer + label-free selector (✅), S4 detection viable (◑), S3 precondition-prediction hard (✗), S5 conformal validity transfers (✅).
• v3 (F1–F4) — research_v3/. F2a alignment surface (✅ synthetic), F1a selection≠scaling (◑), F2b law needs rank-relative-to-background (◑), F3a invariance mechanism (◑), F3 cross-objective decisive (✅ — erosion across DINOv2/supervised/MedDINOv3; MAE not separable).
• v4 (G1–G3) — research_v4/. G1a training-free concentration steering = NEGATIVE (✅ honest): globalization is an entangling transformation, not a removable nuisance — reinforces S2 and sharpens F3. G2 (cross-domain universality) and G3 (deployable tool + benchmark) specced, not built.

Headline numbers (multi-seed where applicable)

• Mid-layer lesion AUROC peak 0.866 ± 0.010 (block 3), final-layer 0.565; label-free tail-gap selector regret 0.006.
• Cross-objective depth-erosion ρ(invariance, AUROC): DINOv2 −0.93, supervised −0.73, MedDINOv3 −0.94.
• The law is exact (gap=(m−r)/m, r*=m, std 0.0 over 40 seeds).
• Coverage-floor ablation: 0.22 (floor) vs 0.82 (membership) small-lesion retention.

What's honest about this body of work

The clean wins (the law, the mid-layer mechanism + selector, the cross-modality method, the conformal certificate) are real and reproducible. The deep extensions (steering, concentration- preserving pretraining via reconstruction) returned negatives that sharpened the science rather than producing new methods — reported as first-class results. The remaining novel lever is G2 (cross-domain universality), a deliberate new project, not a quick eval.

Concurrent work & positioning

FINO (Gardès et al., Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have, arXiv:2606.05107, Meta FAIR, June 2026; code on the DINOv3 FINO branch) adapts vision FMs to scientific domains label-free via metadata-guided SSL training. It is orthogonal: FINO trains/adapts a backbone with metadata; covtoken probes a frozen backbone with pure token geometry — no training, no metadata. It addresses none of covtoken's contributions (depth/layer localization, the rank-vs-concentration law, the conformal certificate, rare-structure focus). Three implications:

• Differentiate in related work: frozen geometric probe + law + certificate, not adaptation.
• F1b lever — TESTED at adapter scale (G1c), not de-risked. No FINO checkpoint is released, so we trained a label-free metadata-guided adapter (the trained counterpart to G1a). Result: SSL-only adaptation erodes localizability (globalization, reproduced by a trained objective), and metadata guidance offsets it directionally (+0.014 mean vs SSL) — but the gap is not CI-significant and it does not recover the depth-eroded concentration subspace toward the mid-layer (0.866). So at adapter scale with anatomical metadata, FINO's lever moves the needle in the right direction but does not solve the erosion. F1b would need full-backbone capacity AND lesion-relevant metadata.
• Complementary experiment, if a FINO checkpoint ships: run the probe on a full FINO-adapted backbone (more capacity than our adapter) — the open question G1c could only lower-bound.

Reproduce

All jobs in jobs/ (PEP-723 uv scripts) run via hf jobs uv run --flavor <t4-medium|t4-small|cpu> --secrets HF_TOKEN -v hf://buckets/Chucks90/eryon-datasets:/mnt <script>. Specs in research_specs/. Figures: paper/figures/ via paper/make_figures.py.