--- license: apache-2.0 tags: - image-classification - probing - frozen-features - imagenet-1k library_name: pytorch --- # Efficient Probing heads **Paper:** [Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency](https://huggingface.co/papers/2506.10178) (ICLR 2026) · [arXiv:2506.10178](https://arxiv.org/abs/2506.10178) · [code & leaderboard](https://github.com/billpsomas/efficient-probing) Trained EP (efficient probing) heads for all 37 frozen encoders of the standing ImageNet-1k benchmark at [billpsomas/efficient-probing](https://github.com/billpsomas/efficient-probing). Each head is a multi-query cross-attention pooling + BatchNorm + linear classifier, trained for 90 epochs with LARS on frozen features. **No backbone weights are included** -- each `config.json` records the exact `--model` / loader flags to rebuild the encoder from its original source. Why two kinds of head: the training code initially kept only a rolling per-epoch checkpoint, so a finished run left its **final** epoch behind rather than its best. 13 early-peaking models were later re-run with best-epoch checkpointing, so their heads are the **peak** and reproduce the leaderboard number exactly. Every file's metadata records both its own accuracy at the saved epoch and the table's best-epoch figure, so nothing has to be taken on trust. ## Loading ```python from huggingface_hub import hf_hub_download import torch path = hf_hub_download("billpsomas/efficient-probing-heads", "dinov3_vit7b/ep_head.pth") ck = torch.load(path, map_location="cpu", weights_only=False) head_state, meta = ck["state_dict"], ck["meta"] # with the benchmark repo on PYTHONPATH: # model = backbones.build_backbone(args, device) # args from meta # probe_heads.build_probe_head(model, args) # model.head.load_state_dict(head_state, strict=True) ``` Or evaluate directly with the benchmark's tool: ``` python tools/eval_reimagenet.py predict \ --head_ckpt ep_head.pth --pred_out preds.json ``` ## Heads | encoder | EP variant | top-1 @ saved epoch | epoch | checkpoint is | |---|---|---:|---:|---| | DINOv3 ViT-7B/16 | ep_all | 88.36 | 6 | peak | | MetaCLIP2 ViT-bigG/14-378 | ep | 88.12 | 6 | peak | | EVA02-CLIP E-14-plus | ep | 87.98 | 6 | peak | | EVA02-CLIP E-14 | ep | 87.70 | 6 | peak | | SigLIP2 SO400M/14 | ep | 87.68 | 6 | peak | | PE-Core L-14/336 | ep | 87.25 | 12 | peak | | MetaCLIP2 ViT-bigG/14 | ep | 87.11 | 6 | peak | | SigLIP2 ViT-L/16 | ep | 87.06 | 6 | peak | | DINOv3 ViT-L/16 | ep_all | 86.73 | 19 | final epoch | | AIMv2 ViT-L/14 | ep | 85.62 | 19 | final epoch | | SigLIP ViT-L/16 | ep | 85.93 | 6 | peak | | DINOv2 ViT-L/14 | ep_all | 85.56 | 15 | peak | | Franca ViT-L/14 | ep_all | 84.28 | 14 | peak | | DINOv3 ViT-B/16 | ep_all | 83.77 | 20 | final epoch | | DINOv2 ViT-B/14 | ep | 83.61 | 25 | final epoch | | RADIO ViT-L/16 | ep | 83.40 | 89 | final epoch | | EVA02 ViT-L/14 | ep | 83.22 | 89 | final epoch | | CLIP ViT-L/14 | ep | 83.22 | 11 | peak | | CAPI ViT-L/14 | ep | 82.43 | 89 | final epoch | | BEiTv2 ViT-B/16 | ep | 81.32 | 89 | final epoch | | RADIO ViT-B/16 | ep | 80.26 | 89 | final epoch | | iBOT ViT-L/16 | ep_all | 79.43 | 89 | final epoch | | Hiera ViT-H/16 | ep | 79.82 | 89 | final epoch | | MAE ViT-L/16 | ep | 79.43 | 89 | final epoch | | I-JEPA ViT-H/14 | ep | 78.80 | 89 | final epoch | | iBOT ViT-B/16 | ep_all | 78.62 | 89 | final epoch | | Hiera ViT-L/16 | ep | 78.51 | 83 | final epoch | | CLIP ViT-B/16 | ep_all | 77.85 | 11 | peak | | DINO ViT-B/16 | ep_all | 77.08 | 89 | final epoch | | MoCov3 ViT-B/16 | ep_all | 76.21 | 89 | final epoch | | Hiera ViT-B/16 | ep | 75.63 | 88 | final epoch | | MAE ViT-B/16 | ep | 75.35 | 86 | final epoch | | MaskFeat ViT-B/16 | ep | 71.68 | 89 | final epoch | | MaskFeat ViT-L/16 | ep | 69.56 | 89 | final epoch | | SimMIM ViT-B/16 | ep | 64.81 | 89 | final epoch | | MAE ViT-S/16 | ep | 64.56 | 89 | final epoch | | DiT DiT-XL/2 | ep | 56.94 | 86 | final epoch | Full provenance (training logs, exact commands, the leaderboard itself) lives in the [GitHub repo](https://github.com/billpsomas/efficient-probing). Heads were trained on ImageNet-1k; use accordingly. ## Citation ```bibtex @inproceedings{psomas2026attention, title = {Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency}, author = {Bill Psomas and Dionysis Christopoulos and Eirini Baltzi and Ioannis Kakogeorgiou and Tilemachos Aravanis and Nikos Komodakis and Konstantinos Karantzalos and Yannis Avrithis and Giorgos Tolias}, booktitle = {The Fourteenth International Conference on Learning Representations}, year = {2026}, url = {https://openreview.net/forum?id=PXo0gtT7Al} } ```