license: mit
pretty_name: Mind the Tail - SAE, Selection Rankings, and Models
tags:
- data-selection
- coreset
- sparse-autoencoder
- dinov3
Mind the Tail: Released Artifacts
This repository contains the artifacts of Mind the Tail: Data Selection Through Sparse Concepts: the sparse autoencoder (SAE) used to extract concepts, the selection rankings of every training pool, and the models trained on the selected subsets.
Paper | Project Page | Code
Structure
├── ckpts/
│ ├── sae/dinov3_b_layer11_in1k_topk32/
│ │ └── sae.pt # TopK SAE, 16,384 concepts
│ └── downstream/
│ ├── nabirds/<ratio>pct/best.pt # ResNet-50
│ ├── stanford_cars/<ratio>pct/best.pt # ResNet-50
│ ├── waterbirds/<ratio>pct/best.pt # ResNet-50
│ ├── spawrious/<ratio>pct/best.pt # ResNet-50
│ └── cc12m/<ratio>pct/clip.pt # CLIP ViT-B/16
└── corsets/
├── nabirds/ranked_examples.parquet
├── stanford_cars/ranked_examples.parquet
├── waterbirds/ranked_examples.parquet
├── spawrious/ranked_examples.parquet
└── cc12m/ranked_examples.parquet
<ratio> is one of 10, 20, 30, 50, and 70.
Sparse Autoencoder
ckpts/sae/dinov3_b_layer11_in1k_topk32/sae.pt is the single SAE used for every dataset: a TopK SAE (k = 32 active concepts per token, 16,384 concepts) trained on 100M layer-11 patch tokens of a frozen DINOv3 ViT-B/16 from ImageNet-1K train. Image-level concept vectors are obtained by max-pooling the patch activations.
The file starts with one JSON line holding the architecture (d_model, d_sae, top_k), followed by a PyTorch state dict with W_enc (768 × 16,384), b_enc, W_dec (16,384 × 768), and b_dec.
Selection Rankings
Each ranked_examples.parquet holds the full ordering of a training pool. A subset at ratio r is the top of this ordering.
Within-class rankings (labelled datasets) have the columns rank_within_class, class_candidate_count, image_path, class_idx, and class_name. The subset at ratio r keeps the top max(1, floor(r · class_candidate_count)) images of every class.
Global ranking (CC12M) has the columns rank_global and image_path. The subset at ratio r keeps the top floor(r · N) images. Paths have the form tar://${DATA_ROOT}/cc12m-wds/<shard>.tar::<key>.jpg, referring to the image of sample <key> in a CC12M WebDataset shard.
All paths are written relative to ${DATA_ROOT}, the folder holding the original datasets, so they resolve against any dataset location.
Models
ResNet-50 (best.pt): ImageNet-pretrained ResNet-50 fine-tuned on the subset of each dataset and ratio (18,700 steps, batch size 128, SGD, lr 1e-3). Each file holds only the weights as {"model_state_dict": ...}. The output classes follow class_idx of the corresponding ranking file.
CLIP ViT-B/16 (clip.pt): trained from scratch on the CC12M subset at each ratio. Each file is a plain state dict of the OpenCLIP ViT-B-16 model.
Datasets
The images themselves are not included; they are available from their original sources: NABirds, Stanford Cars, Waterbirds (waterbird_complete95_forest2water2), Spawrious (one-to-one hard), and CC12M.
Citation
@article{zhang2026mindthetail,
title = {Mind the Tail: Data Selection Through Sparse Concepts},
author = {Zhang, Ziheng and Hulkund, Neha and Lee, Colin and Mai, Zheda and
Shen, Hui and Ding, Ying and Berger-Wolf, Tanya and Beery, Sara and
Chao, Wei-Lun and Gu, Jianyang},
journal = {arXiv preprint arXiv:2610.21007},
year = {2026}
}
License
The artifacts in this repository are released under the MIT License.