File size: 12,379 Bytes

---
license: apache-2.0
library_name: transformers
pipeline_tag: image-feature-extraction
tags:
- clip
- knowledge-distillation
- consensus-distillation
- vit
- safetensors
- custom_code
---

# clip-vitb-mini-distilled

An **8.66M-parameter ViT image encoder** (10.0% of a CLIP-B/16 image
tower) producing 512-d embeddings compatible with the
[CLIP-B/16 LAION-2B](https://huggingface.co/laion/CLIP-ViT-B-16-laion2B-s34B-b88K)
text tower. The primary checkpoint was distilled on **CC12M (10,968,539
images)** against the **generalized-Procrustes consensus of five CLIP
teachers** — never against the deployment teacher — and carries a frozen
512×512 rotation that maps its outputs into the deployment frame, where
it outperforms the student distilled directly against that teacher on
every task gauge, both seeds (full tables below).

## Quick start (AutoModel)

```python
import torch
from transformers import AutoModel, AutoImageProcessor

repo = "AbstractPhil/clip-vitb-mini-distilled"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
proc = AutoImageProcessor.from_pretrained(repo)

px = proc(images=[img], return_tensors="pt")["pixel_values"]
emb = model.get_image_features(px)     # (1, 512), L2-normalized,
                                       # deployment (LAION-B/16) frame
```

The rotation is applied by default (`config.apply_rotation`); pass
`apply_rotation=False` to `get_image_features` for the raw consensus-frame
embedding. Weights are safetensors; the modeling code is in this repo
(`modeling_clip_mini.py`, `configuration_clip_mini.py`).

**Why a rotation exists at all — and when it doesn't.** This model's
consensus target is a generalized-Procrustes *running mean* of five
teachers, which has no privileged frame: the student lands in the
consensus frame rather than any teacher's, so a fitted rotation carries
it the rest of the way. That is a property of how the target was built,
not of consensus distillation. A sibling campaign built its consensus by
mapping every teacher into the frame of one *reference member* of the
set; its frame gap fell monotonically to ~.001 during training and it
ships with no rotation. If you build your own consensus target, pick a
reference member for a pinned frame or a mean for symmetry — and know
which you chose.

**Evaluation-path note:** all published numbers were measured with the
tensor-bicubic evaluation pipeline (interpolate to 160px, CLIP norm — the
path implemented in `loader.py` and `train/dist_bed.py`). On CIFAR-scale
(32px) inputs the shipped `CLIPImageProcessor` path yields 0.5545 vs
0.5708 zero-shot for the same weights, a resize-path difference that
matters only for very small source images; verify against ledger numbers
with the tensor path.

## Repository inventory

| path | content |
|---|---|
| `model.safetensors`, `config.json`, `modeling_clip_mini.py`, `configuration_clip_mini.py`, `preprocessor_config.json` | the champion (consensus composite, seed 1) in AutoModel format, rotation baked in as a buffer |
| `cc12m/*_t88000.pt` | **all 18 CC12M finals** — every arm (InfoNCE, feature-MSE, SigLIP-pairwise, affinity-KL, consensus pure-MSE, consensus composite, gradient-operator/spread/verbatim battery variants), both seeds |
| `cc12m/rotation_s{0,1}.pt` | the frozen deployment rotations (fp32, 512×512, fit metadata included) |
| `cc12m/mid/*_mid.pt` | rolling mid-training snapshots, 15 of the 18 arm-seeds (last 20k-step multiple) — the trajectory record behind the set-point and descent analyses. The three absent ones (feature-MSE s0, InfoNCE s0, affinity-KL s0) trained before mid-checkpointing was added; their finals are complete. |
| `coco/*_t8000.pt` | **all 24 generation-one finals** (COCO budget: the six matrix arms + the six battery arms, both seeds) |
| `student_infonce_s0.pt` | the original generation-one release, kept at its historical path |
| `loader.py` | dependency-free loader for the raw checkpoints (rotation-aware) |
| `train/` | complete training code: bed, objectives, teacher loading/conversion, CC12M data path, consensus-target builder, reproduction guide (`train/TRAINING.md`) |
| `ledgers/*.jsonl` | per-run ledgers, both generations (COCO rows `steps=8000`, CC12M rows `steps=88000`; `eval_only` rows carry CIFAR-100) |
| `ledgers/frame_check_pod_s{0,1}.json` | rotation forensics, all arms, both seeds |
| `ledgers/rotated_eval_*.json` | full rotated evaluations (champion + pure-MSE consensus) |
| `article_cc12m_distillation.md` | the campaign report |

## Architecture

| field | value |
|---|---|
| type | ViT, CLS-token readout, linear projection head |
| hidden size / depth / heads | 240 / 12 / 4 |
| patch / input | 16 / 160×160 |
| projection | 512-d (deployment teacher's projection space) |
| parameters | 8,664,752 total (8,541,360 trunk + 123,392 head) |
| rotation buffer | 512×512 orthogonal, frozen; fitted post-hoc on 2,500 COCO-val pairs (fp64 orthogonal Procrustes), maps consensus frame → LAION-B/16 frame |
| preprocessing | shorter edge → 182 (bicubic), center-crop 160, CLIP mean/std |

## Results

**Headline — champion (rotated) vs the directly-distilled student, CC12M,
seed 0 / seed 1:**

| gauge | consensus composite + rotation | InfoNCE vs deployment teacher | teacher | random floor |
|---|---|---|---|---|
| zero-shot CIFAR-10 | **.5412 / .5708** | .5208 / .4979 | .946 | .0996 |
| zero-shot CIFAR-100 | **.2210 / .2487** | .1958 / .2092 | .759 | .0068 |
| COCO R@1 (i→t) | **.1322 / .1360** | .1252 / .1178 | .417 | .000 |
| COCO R@5 | **.3140 / .3300** | .2968 / .2910 | .669 | .001 |

Champion capability: 44.7% / 48.1% of the teacher's zero-shot mean
(CIFAR-10+100) at 10% of its image-tower parameters. Scaling arc across
data generations: 21.2% (118k images) → 42.0/41.5% (11M, direct InfoNCE)
→ 44.7/48.1% (11M, consensus + rotation).

**Full CC12M matrix** (nine objectives × two seeds, 88k steps ≈ 2.05
epochs; `agree` = cosine to deployment teacher unless *(own)* = the arm's
own target):

| objective | zs C10 | zs C100 | R@1 | R@5 | agree | eff. rank |
|---|---|---|---|---|---|---|
| InfoNCE (deployment teacher) | .5208/.4979 | .1958/.2092 | .1252/.1178 | .2968/.2910 | .543/.542 | 188/188 |
| feature MSE (mimicry) | .4429/.4826 | .1568/.1656 | .0760/.0716 | .2074/.1996 | .735/.738 | 154/153 |
| SigLIP-pairwise (768-d, own tower) | .4534/.4573 | .1625/.1702 | .0904/.0914 | .2278/.2278 | .742/.743 *(own)* | 155/154 |
| consensus composite (NCE+MSE) | .3271/.3928 | .1213/.1254 | .0344/.0374 | .1180/.1178 | .741/.742 *(own)* | 172/173 |
| + gradient operator | .3398/.3315 | .1181/.1335 | .0374/.0352 | .1160/.1154 | .742/.742 *(own)* | 173/172 |
| + spread force (1e-3) | .2819/.3370 | .1144/.1172 | .0386/.0328 | .1230/.1118 | .743/.741 *(own)* | 173/174 |
| verbatim ancestor recipe | .3188/.3616 | .1269/.1377 | .0554/.0506 | .1582/.1512 | .531/.530 | 185/185 |
| consensus pure-MSE | .2738/.2681 | .0575/.0567 | .0114/.0100 | .0410/.0458 | .814/.810 *(own)* | 141/142 |
| affinity (similarity-KL) | .1076/.1035 | .0117/.0105 | .0000/.0008 | .0012/.0022 | −.011/−.017 | 103/89 |

**Rotation forensics** (agree→deployment pre → post; zs C10 pre → post;
both seeds):

| arm | agree | zs |
|---|---|---|
| consensus composite | .304→.732 / .302→.731 | .327→.541 / .393→.571 |
| consensus pure-MSE | .297→.765 / .293→.763 | .274→.434 / .268→.394 |
| verbatim recipe | .258→.644 / .258→.642 | .319→.487 / .362→.532 |
| + gradient operator | .302→.731 / .304→.730 | .340→.517 / .332→.568 |
| + spread force | .304→.731 / .304→.728 | .282→.485 / .337→.489 |
| affinity | −.011→.304 / −.017→.301 | .108→.138 / .104→.132 |
| InfoNCE (control) | .543→.633 / .542→.631 | .521→.473 / .498→.503 |

Note the pure-MSE consensus row: rotated agreement .765/.763 *exceeds*
direct mimicry's in-frame .735/.738 — the consensus target carries the
deployment teacher's geometry better than direct distillation preserves
it — but without a contrastive term it cannot retrieve (rotated R@1
.046/.051). The composite's two terms are both load-bearing.

**Generation one (COCO, 118k images, 8k steps), for the scale
comparison:** mimicry .2693/.2657 zs vs InfoNCE .2668/.2308 (mimicry won
coarse zero-shot at small scale; the verdict reversed at 11M); InfoNCE
retrieval .0580/.0544 vs .0248/.0262; consensus raw .1543/.1609 — its
loss to single-teacher distillation was preregistered and refuted 2/2 at
this scale before reversing at CC12M with the rotation. Full
generation-one ledgers included.

## Training

Everything needed to reproduce any checkpoint is in `train/`
(`train/TRAINING.md` is the step-by-step guide). Summary:

| item | value |
|---|---|
| data (gen 2) | CC12M, 10,968,539 images via [pixparse/cc12m-wds](https://huggingface.co/datasets/pixparse/cc12m-wds); read by tar-offset index, no extraction |
| targets | precomputed teacher features, [bulk-cc12m-features](https://huggingface.co/datasets/AbstractPhil/bulk-cc12m-features) (fp16, unnormalized; L2-normalized per batch on GPU) |
| champion objective | symmetric InfoNCE (t=0.07) + per-sample MSE, against the 5-teacher GPA consensus mean |
| consensus target | generalized Procrustes mean of clip_b16_laion2b, clip_b32_openai, clip_b32_laion2b, clip_b32_datacomp, clip_b16_openai (5 iterations, fp64 alignment SVD on a fixed 10k subsample; converged mean alignment ≈ 0.91) |
| optimizer | Adam 3e-4, weight decay 0, fp32, TF32 off (exception: the verbatim replication arm uses its source recipe — AdamW 0.01, warmup+cosine, clip 1.0) |
| schedule | 88,000 steps, batch 256 (≈ 2.05 epochs); gen-1: 8,000 steps on COCO (≈ 17.3 epochs) |
| seeds | student and data order CRC-derived from arm name + seed index; two seeds per arm |
| rotation | fitted after training: fp64 orthogonal Procrustes, student→teacher, on the fixed 2,500-pair COCO-val split; frozen thereafter |
| hardware | one 24 GB GPU per run (peak 4.83 GB); ~8 h per CC12M run on an A40, CPU-decode bound |

## Evaluation protocol

CIFAR-10/100 zero-shot via the LAION-B/16 text tower ("a photo of a
{class}"), images tensor-upscaled (bicubic) to the student's 160px input;
teacher judged at its native 224px in the same harness; floor (random
init) and ceiling (teacher) bracket every table. COCO retrieval:
image→text R@1/R@5 over the 5,000 val images against first-caption text
embeddings, paired protocol. Agreement: mean cosine to the arm's target
embeddings on COCO val. Effective rank and a spread statistic (CV of
Cayley-Menger 4-simplex volumes on a fixed 16-d projection) are read-only
health instruments. Every evaluation is deterministic: `--eval-only`
re-runs reproduce ledger rows to four decimal places, and this release
was gated by fresh tokenless downloads reproducing the headline numbers
exactly (raw-loader path and AutoModel path, bit-parity verified).

## Limitations

Research model. 48% of teacher zero-shot at best; the preregistered 60%
viability bar is unmet (the data-scaling curve is unbent — see the
article). CIFAR absolute numbers are harness-scoped (32px upscale; see
the evaluation-path note). Two seeds per objective; one batch size
(256); one student size; one consensus family; one deployment frame.
The rotation fit consumes 2,500 labeled pairs of the deployment frame.

## Related

- Campaign report: `article_cc12m_distillation.md` (in this repo)
- Teacher banks: [bulk-cc12m-features](https://huggingface.co/datasets/AbstractPhil/bulk-cc12m-features) · [bulk-coco-features](https://huggingface.co/datasets/AbstractPhil/bulk-coco-features)
- Memory-bank line: [geolip-vit-captionbank-coco](https://huggingface.co/AbstractPhil/geolip-vit-captionbank-coco)
- Objective genealogy: [loss-manifest](https://huggingface.co/AbstractPhil/loss-manifest)

## References

- Radford et al., CLIP — https://arxiv.org/abs/2103.00020
- van den Oord et al., InfoNCE — https://arxiv.org/abs/1807.03748
- Yang et al., CLIP-KD — https://arxiv.org/abs/2307.12732
- Wu et al., TinyCLIP — https://arxiv.org/abs/2309.12314
- Zhai et al., SigLIP — https://arxiv.org/abs/2303.15343
- Gower, Generalized Procrustes Analysis — https://doi.org/10.1007/BF02291478
- Changpinyo et al., Conceptual 12M — https://arxiv.org/abs/2102.08981
- Cherti et al., OpenCLIP scaling — https://arxiv.org/abs/2212.07143