--- license: apache-2.0 library_name: timm pipeline_tag: image-classification datasets: - ILSVRC/imagenet-1k metrics: - accuracy tags: - vision-transformer - image-classification - knowledge-distillation - feature-distillation - model-compression - interpretability - explainability - representation-analysis - encoding-mismatch - spectral-energy-pattern - spectral-kd - low-rank - pca - svd - deit-tiny - distilled-deit-tiny - cait-s24 - lift - widelast - pytorch - timm - icml-2026 - arxiv:2511.15572 model-index: - name: Lift results: - task: type: image-classification name: Image Classification dataset: type: ILSVRC/imagenet-1k name: ImageNet-1K split: validation metrics: - type: accuracy name: Top-1 Accuracy (%) value: 77.53 source: name: ICML 2026 paper, Table 2 url: https://arxiv.org/abs/2511.15572 - name: WideLast results: - task: type: image-classification name: Image Classification dataset: type: ILSVRC/imagenet-1k name: ImageNet-1K split: validation metrics: - type: accuracy name: Top-1 Accuracy (%) value: 78.23 source: name: ICML 2026 paper, Table 2 url: https://arxiv.org/abs/2511.15572 --- # From Per-Image Low-Rank to Encoding Mismatch This repository releases the **Lift** and **WideLast** checkpoints from the ICML 2026 paper: > **From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformers** > Huiyuan Tian, Bonan Xu, and Shijian Li The paper studies why feature distillation between heterogeneous Vision Transformers can remain difficult even when individual feature maps are strongly low-rank. Its analysis distinguishes: - **Per-image low-rank structure**, measured by sample-wise singular value decomposition (SVD). - **Dataset-level representational structure**, measured by PCA over features from many images. - **Spectral Energy Patterns (SEP)**, which characterize how feature energy is distributed across channels. The central finding is an **encoding mismatch**: compact students and wider teachers can encode dataset-level information using substantially different channel-space geometries. Two simple architecture modifications reduce this mismatch: - **Lift** retains a linear endpoint projector that lifts the student's final representation from 192 to 384 channels. - **WideLast** widens only the last transformer block from 192 to 384 channels. ## Links - [Paper on arXiv](https://arxiv.org/abs/2511.15572) - [Paper on OpenReview](https://openreview.net/forum?id=2Ud1nkQrVZ) - [Official code, analysis scripts, raw numeric data, and training recipes](https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch) - [ICML 2026 poster](https://huggingface.co/Huiyuancs/Encoding_Mismatch/blob/main/icml_poster.pdf) ## Released Checkpoints | Checkpoint | File | Architecture | Distillation recipe | ImageNet-1K top-1 | | ---------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------- | ----------------: | | Lift | [`models/Lift/pytorch_model.pth`](https://huggingface.co/Huiyuancs/Encoding_Mismatch/blob/main/models/Lift/pytorch_model.pth) | DeiT-Tiny-derived student with a retained 192→384 endpoint projector | SoftKD + SpectralKD, CaiT-S24 teacher | **77.53%** | | WideLast | [`models/WideLast/pytorch_model.pth`](https://huggingface.co/Huiyuancs/Encoding_Mismatch/blob/main/models/WideLast/pytorch_model.pth) | DeiT-Tiny-derived student with blocks 1–11 at 192 dimensions and block 12 at 384 dimensions | SoftKD + MSE, CaiT-S24 teacher | **78.23%** | The reported baseline distilled DeiT-Tiny accuracy is **74.86%** under the paper's experimental protocol. ## Model Details ### Shared setting - **Task:** ImageNet-1K image classification - **Input resolution:** 224 × 224 - **Student family:** DeiT-Tiny-derived Vision Transformer - **Teacher:** CaiT-S24 - **Framework:** PyTorch and `timm` - **Training duration:** 300 epochs - **Primary metric:** ImageNet-1K validation top-1 accuracy These models use custom architectures and checkpoint-loading code from the accompanying GitHub repository. They are **not** packaged as `transformers.AutoModel` checkpoints and are not expected to run directly through the Hugging Face hosted inference widget. ### Lift Lift preserves the standard 192-dimensional DeiT-Tiny transformer blocks and adds a learned linear projection at the student endpoint: $$ \mathbb{R}^{192} \rightarrow \mathbb{R}^{384}. $$ Unlike a training-only projector, this layer is retained at inference. The released Lift checkpoint corresponds to the **SoftKD + SpectralKD** result reported in Table 2 of the paper. ### WideLast WideLast keeps the first eleven transformer blocks at 192 channels and widens only the final block to 384 channels. The final block uses six attention heads, while the preceding 192-dimensional blocks use three heads. The released WideLast checkpoint corresponds to the **SoftKD + MSE** result reported in Table 2. ## Evaluation Results ### Main ImageNet-1K results | Student | Distillation objective | Top-1 accuracy | | ---------------------------- | ----------------------- | -------------: | | Distilled DeiT-Tiny baseline | Original baseline | 74.86% | | Lift | SoftKD | 77.23% | | Lift | SoftKD + MSE | 77.50% | | **Lift** | **SoftKD + SpectralKD** | **77.53%** | | WideLast | SoftKD | 77.88% | | WideLast | SoftKD + SpectralKD | 78.16% | | **WideLast** | **SoftKD + MSE** | **78.23%** | ### Inference cost | Model | Parameters | Change vs. baseline | FLOPs | Change vs. baseline | | ---------------------------- | ---------: | ------------------: | -----: | ------------------: | | Distilled DeiT-Tiny baseline | 5,717,416 | — | 2.507G | — | | Lift | 5,983,528 | +4.65% | 2.536G | +1.17% | | WideLast | 7,239,016 | +26.61% | 3.089G | +23.20% | Results are taken from the paper. Reproduction can vary with software versions, distributed-training configuration, data access, and numerical nondeterminism. ## Installation Clone the implementation repository and install its dependencies: ```bash git clone https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch.git cd From-Per-Image-Low-Rank-to-Encoding-Mismatch conda create -n encoding-mismatch python=3.10 -y conda activate encoding-mismatch pip install -r requirements.txt pip install huggingface_hub ``` Download this Hugging Face repository: ```python from huggingface_hub import snapshot_download snapshot_download( repo_id="Huiyuancs/Encoding_Mismatch", local_dir="./Encoding_Mismatch", ) ``` Expected checkpoint paths: ```text Encoding_Mismatch/ └── models/ ├── Lift/ │ └── pytorch_model.pth └── WideLast/ └── pytorch_model.pth ``` ## Data Preparation The evaluation and training scripts expect ImageNet-1K in the standard `ImageFolder` layout: ```text /path/to/ILSVRC/ ├── train/ └── val/ ``` ImageNet is not redistributed with this repository. Users must obtain it under the dataset's applicable terms. ## Evaluate the Released Checkpoints Run these commands from the cloned GitHub repository. ### Lift ```bash cd training/lift python main.py \ --eval \ --resume /absolute/path/to/Encoding_Mismatch/models/Lift/pytorch_model.pth \ --data-path /path/to/ILSVRC \ --model deit_tiny_patch16_224 \ --batch-size 256 \ --num_workers 8 \ --use-modified-student \ --expansion-start-layer 11 \ --expansion-type step \ --expansion-use-ln \ --expansion-target-dim 384 ``` ### WideLast ```bash cd training/widelast python main.py \ --eval \ --resume /absolute/path/to/Encoding_Mismatch/models/WideLast/pytorch_model.pth \ --data-path /path/to/ILSVRC \ --model deit_tiny_patch16_224 \ --batch-size 256 \ --num_workers 8 \ --custom-arch \ --arch-schedule heads_step ``` The repository's evaluation code expects a PyTorch checkpoint dictionary containing a `model` state dictionary. ## Reproduce Training Set the dataset and distributed-training configuration: ```bash export DATA_PATH=/path/to/ILSVRC export GPUS=0,1,2,3 export NPROC=4 ``` Train the released Lift recipe: ```bash cd training/lift bash scripts/run_cait_softkd_spectralkd.sh ``` Train the released WideLast recipe: ```bash cd training/widelast bash scripts/run_cait_softkd_mse.sh ``` Additional ablation recipes are available in the `training/lift/scripts/` and `training/widelast/scripts/` directories. ## Analysis Artifacts and Raw Data The official GitHub repository includes: - sample-wise SVD analysis; - dataset-level PCA analysis; - Spectral Energy Pattern analysis; - SEP permutation-robustness analysis; - scripts for regenerating figures and tables; - prepared `.npz` and `.csv` numeric results in [`Raw data`](https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch/tree/main/Raw%20data). See the [ICML 2026 poster](https://huggingface.co/Huiyuancs/Encoding_Mismatch/blob/main/icml_poster.pdf) for a compact visual overview. ## Intended Use The checkpoints are intended for: - research on feature and knowledge distillation in Vision Transformers; - analysis of teacher–student representation mismatch; - controlled ImageNet-1K evaluation of Lift and WideLast; - studies of low-rank structure, dataset-level subspaces, and spectral channel statistics; - reproduction and extension of the ICML 2026 paper. ## Citation ```bibtex @inproceedings{tian2026encodingmismatch, title = {From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformers}, author = {Tian, Huiyuan and Xu, Bonan and Li, Shijian}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, year = {2026} } ``` ## License This Hugging Face repository is marked as **Apache-2.0**. Users are responsible for complying with the licenses and terms of the code dependencies, ImageNet-1K, and any upstream assets.