File size: 1,821 Bytes
1161ec2 d191b8e 1161ec2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | ---
license: mit
pipeline_tag: video-classification
tags:
- video
- action-recognition
- efficient-inference
- token-selection
library_name: pytorch
---
# LookWhen ViT-B/16
Pre-trained checkpoint for **"LookWhen? Fast Video Recognition by Learning When, Where,
and What to Compute"** ([arXiv:2605.06809](https://arxiv.org/abs/2605.06809)).
LookWhen splits video recognition in two: a shallow **selector** scores every patch of a
2x-downscaled (2x in each dimension) clip, and a deep **extractor** processes only the top-K of them while still
predicting features of the whole video.
This is the model used throughout the paper: ViT-B/16, pre-trained for 20 epochs on
Kinetics-400 and SSv2 by distilling InternVideo2's video token and DINOv3's frame and
patch tokens, with selection supervised by top1-distance token uniqueness. It was trained
over sparsity 70-95%, so it runs anywhere in that range without retraining.
## Usage
```bash
pip install git+https://github.com/alisalamatian1/LookWhen.git
```
```python
import torch
from lookwhen import LookWhen
from lookwhen.data.video import load_clip
model = LookWhen.from_pretrained().cuda().eval() # downloads this checkpoint
clip = load_clip("video.mp4", num_frames=16, img_size=224).unsqueeze(0).cuda()
out = model(clip, keep_ratio=0.1) # keep_ratio = 1 - sparsity
out["video"] # (1, 768) video embedding
```
Code, training, and evaluation: https://github.com/alisalamatian1/LookWhen
## Citation
```bibtex
@article{salamatian2026lookwhen,
title = {LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute},
author = {Salamatian, Ali and Fuller, Anthony and Sarkar, Pritam and
Green, James R. and Sigal, Leonid and Shelhamer, Evan},
journal = {arXiv preprint arXiv:2605.06809},
year = {2026}
}
```
|