---
license: apache-2.0
pipeline_tag: image-classification
---

# LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation

[Paper](https://huggingface.co/papers/2606.19483) | [GitHub](https://github.com/KevinZ0217/LEAP) | [Project Page](https://kevinz0217.github.io/LEAP_page/)

This repository contains the ViT-Tiny and ViT-S checkpoints (No Register) distilled from ViT-G DINOv2 on ImageNet-100 and ImageNet-1K. The knowledge distillation process follows the procedure proposed in the paper **"LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation"**.

### Introduction

Vision Foundation Models (VFMs) with ViT backbones, such as DINOv2, are computationally demanding. LEAP (Layer-skipping Efficiency via Adaptive Progression) is a training curriculum for ViT feature-based knowledge distillation. Instead of supervising the student against a fixed teacher block, LEAP advances the supervisory target through the teacher's feature maps (shallow-to-deep) based on online CKA alignment. This allows the student to build a foundational representation before tackling higher-level abstractions.

### Use cases
The ViT models output feature maps that can be used for a variety of downstream tasks, including:
- Image Classification
- Instance Retrieval
- Semantic Segmentation

### Performance

#### ImageNet-100:
![image](https://cdn-uploads.huggingface.co/production/uploads/63d84d163130cadcaf8a976a/MSp0sMCOEnXsdKry5wRvF.png)

![image](https://cdn-uploads.huggingface.co/production/uploads/63d84d163130cadcaf8a976a/wSlK06UTvnlRY5o4vvpN_.png)

![image](https://cdn-uploads.huggingface.co/production/uploads/63d84d163130cadcaf8a976a/36r7mqVc-Qwyd-B_gRJ9p.png)

#### ImageNet-1K:

![image](https://cdn-uploads.huggingface.co/production/uploads/63d84d163130cadcaf8a976a/UwmTdXFTrmWeWl_33P2dP.png)

![image](https://cdn-uploads.huggingface.co/production/uploads/63d84d163130cadcaf8a976a/0grYPjq1UDpGv8zh6GCi3.png)

![image](https://cdn-uploads.huggingface.co/production/uploads/63d84d163130cadcaf8a976a/u9MVacA5A284XEgRGuKe5.png)

### Citation

```bibtex
@article{leap2026,
  title={LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation},
  author={Zhang, Jiaqi and Lee, Ashton and Wong, Anthony and Zou, John and BuGhanem, Sami and Balestriero, Randall},
  journal={arXiv preprint arXiv:2606.19483},
  year={2026}
}
```