Update README.md

README.md

@@ -1,6 +1,306 @@
 ---
 license: apache-2.0
-papers:
-  - 2511.15572
+library_name: timm
+pipeline_tag: image-classification
+datasets:
+- ILSVRC/imagenet-1k
+metrics:
+- accuracy
+tags:
+- vision-transformer
+- image-classification
+- knowledge-distillation
+- feature-distillation
+- model-compression
+- interpretability
+- explainability
+- representation-analysis
+- encoding-mismatch
+- spectral-energy-pattern
+- spectral-kd
+- low-rank
+- pca
+- svd
+- deit-tiny
+- distilled-deit-tiny
+- cait-s24
+- lift
+- widelast
+- pytorch
+- timm
+- icml-2026
+- arxiv:2511.15572
+model-index:
+- name: Lift
+  results:
+  - task:
+      type: image-classification
+      name: Image Classification
+    dataset:
+      type: ILSVRC/imagenet-1k
+      name: ImageNet-1K
+      split: validation
+    metrics:
+    - type: accuracy
+      name: Top-1 Accuracy (%)
+      value: 77.53
+    source:
+      name: ICML 2026 paper, Table 2
+      url: https://arxiv.org/abs/2511.15572
+- name: WideLast
+  results:
+  - task:
+      type: image-classification
+      name: Image Classification
+    dataset:
+      type: ILSVRC/imagenet-1k
+      name: ImageNet-1K
+      split: validation
+    metrics:
+    - type: accuracy
+      name: Top-1 Accuracy (%)
+      value: 78.23
+    source:
+      name: ICML 2026 paper, Table 2
+      url: https://arxiv.org/abs/2511.15572
 ---
 
+# From Per-Image Low-Rank to Encoding Mismatch
+
+This repository releases the **Lift** and **WideLast** checkpoints from the ICML 2026 paper:
+
+> **From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformers**  
+> Huiyuan Tian, Bonan Xu, and Shijian Li
+
+The paper studies why feature distillation between heterogeneous Vision Transformers can remain difficult even when individual feature maps are strongly low-rank. Its analysis distinguishes:
+
+- **Per-image low-rank structure**, measured by sample-wise singular value decomposition (SVD).
+- **Dataset-level representational structure**, measured by PCA over features from many images.
+- **Spectral Energy Patterns (SEP)**, which characterize how feature energy is distributed across channels.
+
+The central finding is an **encoding mismatch**: compact students and wider teachers can encode dataset-level information using substantially different channel-space geometries. Two simple architecture modifications reduce this mismatch:
+
+- **Lift** retains a linear endpoint projector that lifts the student's final representation from 192 to 384 channels.
+- **WideLast** widens only the last transformer block from 192 to 384 channels.
+
+## Links
+
+- [Paper on arXiv](https://arxiv.org/abs/2511.15572)
+- [Paper on OpenReview](https://openreview.net/forum?id=2Ud1nkQrVZ)
+- [Official code, analysis scripts, raw numeric data, and training recipes](https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch)
+- [ICML 2026 poster](https://huggingface.co/Huiyuancs/Encoding_Mismatch/blob/main/icml_poster.pdf)
+
+## Released Checkpoints
+
+| Checkpoint | File                                                         | Architecture                                                 | Distillation recipe                   | ImageNet-1K top-1 |
+| ---------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------- | ----------------: |
+| Lift       | [`models/Lift/pytorch_model.pth`](https://huggingface.co/Huiyuancs/Encoding_Mismatch/blob/main/models/Lift/pytorch_model.pth) | DeiT-Tiny-derived student with a retained 192→384 endpoint projector | SoftKD + SpectralKD, CaiT-S24 teacher |        **77.53%** |
+| WideLast   | [`models/WideLast/pytorch_model.pth`](https://huggingface.co/Huiyuancs/Encoding_Mismatch/blob/main/models/WideLast/pytorch_model.pth) | DeiT-Tiny-derived student with blocks 1–11 at 192 dimensions and block 12 at 384 dimensions | SoftKD + MSE, CaiT-S24 teacher        |        **78.23%** |
+
+The reported baseline distilled DeiT-Tiny accuracy is **74.86%** under the paper's experimental protocol.
+
+## Model Details
+
+### Shared setting
+
+- **Task:** ImageNet-1K image classification
+- **Input resolution:** 224 × 224
+- **Student family:** DeiT-Tiny-derived Vision Transformer
+- **Teacher:** CaiT-S24
+- **Framework:** PyTorch and `timm`
+- **Training duration:** 300 epochs
+- **Primary metric:** ImageNet-1K validation top-1 accuracy
+
+These models use custom architectures and checkpoint-loading code from the accompanying GitHub repository. They are **not** packaged as `transformers.AutoModel` checkpoints and are not expected to run directly through the Hugging Face hosted inference widget.
+
+### Lift
+
+Lift preserves the standard 192-dimensional DeiT-Tiny transformer blocks and adds a learned linear projection at the student endpoint:
+
+$$
+\mathbb{R}^{192} \rightarrow \mathbb{R}^{384}.
+$$
+
+Unlike a training-only projector, this layer is retained at inference. The released Lift checkpoint corresponds to the **SoftKD + SpectralKD** result reported in Table 2 of the paper.
+
+### WideLast
+
+WideLast keeps the first eleven transformer blocks at 192 channels and widens only the final block to 384 channels. The final block uses six attention heads, while the preceding 192-dimensional blocks use three heads. The released WideLast checkpoint corresponds to the **SoftKD + MSE** result reported in Table 2.
+
+## Evaluation Results
+
+### Main ImageNet-1K results
+
+| Student                      | Distillation objective  | Top-1 accuracy |
+| ---------------------------- | ----------------------- | -------------: |
+| Distilled DeiT-Tiny baseline | Original baseline       |         74.86% |
+| Lift                         | SoftKD                  |         77.23% |
+| Lift                         | SoftKD + MSE            |         77.50% |
+| **Lift**                     | **SoftKD + SpectralKD** |     **77.53%** |
+| WideLast                     | SoftKD                  |         77.88% |
+| WideLast                     | SoftKD + SpectralKD     |         78.16% |
+| **WideLast**                 | **SoftKD + MSE**        |     **78.23%** |
+
+### Inference cost
+
+| Model                        | Parameters | Change vs. baseline |  FLOPs | Change vs. baseline |
+| ---------------------------- | ---------: | ------------------: | -----: | ------------------: |
+| Distilled DeiT-Tiny baseline |  5,717,416 |                   — | 2.507G |                   — |
+| Lift                         |  5,983,528 |              +4.65% | 2.536G |              +1.17% |
+| WideLast                     |  7,239,016 |             +26.61% | 3.089G |             +23.20% |
+
+Results are taken from the paper. Reproduction can vary with software versions, distributed-training configuration, data access, and numerical nondeterminism.
+
+## Installation
+
+Clone the implementation repository and install its dependencies:
+
+```bash
+git clone https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch.git
+cd From-Per-Image-Low-Rank-to-Encoding-Mismatch
+
+conda create -n encoding-mismatch python=3.10 -y
+conda activate encoding-mismatch
+pip install -r requirements.txt
+pip install huggingface_hub
+```
+
+Download this Hugging Face repository:
+
+```python
+from huggingface_hub import snapshot_download
+
+snapshot_download(
+    repo_id="Huiyuancs/Encoding_Mismatch",
+    local_dir="./Encoding_Mismatch",
+)
+```
+
+Expected checkpoint paths:
+
+```text
+Encoding_Mismatch/
+└── models/
+    ├── Lift/
+    │   └── pytorch_model.pth
+    └── WideLast/
+        └── pytorch_model.pth
+```
+
+## Data Preparation
+
+The evaluation and training scripts expect ImageNet-1K in the standard `ImageFolder` layout:
+
+```text
+/path/to/ILSVRC/
+├── train/
+└── val/
+```
+
+ImageNet is not redistributed with this repository. Users must obtain it under the dataset's applicable terms.
+
+## Evaluate the Released Checkpoints
+
+Run these commands from the cloned GitHub repository.
+
+### Lift
+
+```bash
+cd training/lift
+
+python main.py \
+  --eval \
+  --resume /absolute/path/to/Encoding_Mismatch/models/Lift/pytorch_model.pth \
+  --data-path /path/to/ILSVRC \
+  --model deit_tiny_patch16_224 \
+  --batch-size 256 \
+  --num_workers 8 \
+  --use-modified-student \
+  --expansion-start-layer 11 \
+  --expansion-type step \
+  --expansion-use-ln \
+  --expansion-target-dim 384
+```
+
+### WideLast
+
+```bash
+cd training/widelast
+
+python main.py \
+  --eval \
+  --resume /absolute/path/to/Encoding_Mismatch/models/WideLast/pytorch_model.pth \
+  --data-path /path/to/ILSVRC \
+  --model deit_tiny_patch16_224 \
+  --batch-size 256 \
+  --num_workers 8 \
+  --custom-arch \
+  --arch-schedule heads_step
+```
+
+The repository's evaluation code expects a PyTorch checkpoint dictionary containing a `model` state dictionary.
+
+## Reproduce Training
+
+Set the dataset and distributed-training configuration:
+
+```bash
+export DATA_PATH=/path/to/ILSVRC
+export GPUS=0,1,2,3
+export NPROC=4
+```
+
+Train the released Lift recipe:
+
+```bash
+cd training/lift
+bash scripts/run_cait_softkd_spectralkd.sh
+```
+
+Train the released WideLast recipe:
+
+```bash
+cd training/widelast
+bash scripts/run_cait_softkd_mse.sh
+```
+
+Additional ablation recipes are available in the `training/lift/scripts/` and `training/widelast/scripts/` directories.
+
+## Analysis Artifacts and Raw Data
+
+The official GitHub repository includes:
+
+- sample-wise SVD analysis;
+- dataset-level PCA analysis;
+- Spectral Energy Pattern analysis;
+- SEP permutation-robustness analysis;
+- scripts for regenerating figures and tables;
+- prepared `.npz` and `.csv` numeric results in [`Raw data`](https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch/tree/main/Raw%20data).
+
+See the [ICML 2026 poster](https://huggingface.co/Huiyuancs/Encoding_Mismatch/blob/main/icml_poster.pdf) for a compact visual overview.
+
+## Intended Use
+
+The checkpoints are intended for:
+
+- research on feature and knowledge distillation in Vision Transformers;
+- analysis of teacher–student representation mismatch;
+- controlled ImageNet-1K evaluation of Lift and WideLast;
+- studies of low-rank structure, dataset-level subspaces, and spectral channel statistics;
+- reproduction and extension of the ICML 2026 paper.
+
+## Citation
+
+```bibtex
+@inproceedings{tian2026encodingmismatch,
+  title     = {From Per-Image Low-Rank to Encoding Mismatch:
+               Rethinking Feature Distillation in Vision Transformers},
+  author    = {Tian, Huiyuan and Xu, Bonan and Li, Shijian},
+  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
+  year      = {2026}
+}
+```
+
+## License
+
+This Hugging Face repository is marked as **Apache-2.0**. Users are responsible for complying with the licenses and terms of the code dependencies, ImageNet-1K, and any upstream assets.