AbstractPhil
/

clip-vitb-mini-distilled

@@ -5,6 +5,7 @@ pipeline_tag: image-feature-extraction
 tags:
 - clip
 - knowledge-distillation
 - infonce
 - vit
 - research
@@ -12,63 +13,51 @@ tags:
 # clip-vitb-mini-distilled
-An **8.66M-parameter ViT-Ti-class student** distilled from **CLIP ViT-B/16
-(LAION-2B)** pooled image features on **COCO-2017 train only** (118,287
-images, ~17 epochs) — a controlled, budget-scoped experiment in CLIP
-distillation objectives, shipped with its full ledger set and a negative
-headline honestly stated.
-**This is a research artifact, not a production model.** At this data budget
-the student reaches **~21% of the teacher's zero-shot capability** (CIFAR-10 +
-CIFAR-100 mean). The interesting cargo is the *comparative loss study* and the
-**frame-ambiguity finding** below.
-## Headline numbers (test-time, deterministic eval)
-| model | zs CIFAR-10 | zs CIFAR-100 | COCO R@1 (i→t) | params |
 |---|---|---|---|---|
-| teacher CLIP-B/16 LAION-2B | 0.946 | 0.759 | 0.417 | 86M (image tower) |
-| **this student (InfoNCE, seed 0)** | **0.267** | **0.094** | **0.058** | **8.66M** |
-| random-init floor | 0.100 | 0.007 | 0.000 | 8.66M |
-Zero-shot uses the **teacher's text tower** (the student lives in the
-teacher's 512-d projection space). CIFAR images are bicubic-upscaled to the
-student's 160px input; the teacher is judged at its native 224px in the same
-harness.
-## The loss matrix (6 objectives × 2 seeds, 8000 steps, batch 256)
-| arm | zs C10 (s0/s1) | zs C100 | COCO R@1 | teacher-agree | eff. rank |
-|---|---|---|---|---|---|
-| feature MSE (mimicry) | .269 / .266 | .062 | .025 / .026 | **.714 / .718** | 122 |
-| **InfoNCE (ship arm)** | .267 / .231 | **.094 / .085** | **.058 / .054** | .404 / .401 | **184** |
-| SigLIP pairwise | .221 / .253 | — | .043 / .040 | — | 150 |
-| affinity KL | .109 / .111 | — | .000 | −.02 (see below) | 175 |
-| InfoNCE + geometric terms | .248 / .253 | .084 / .072 | .057 / .051 | .404 | 185 |
-| 5-teacher consensus MSE | .154 / .161 | — | .004 / .006 | .29 (see below) | 108 |
-Findings, each 2/2 seeds unless noted:
-- **The gauge split.** Feature-MSE mimicry wins *coarse* zero-shot and
-  teacher agreement; InfoNCE wins *fine-grained* zero-shot (+3.2/+2.2 pts
-  CIFAR-100), retrieval (2.2×), and effective rank (184 vs 122). Pick your
-  objective by the gauge you will be judged on.
-- **Frame ambiguity (the campaign's main finding).** The affinity-KL arm
-  scores at floor on every absolute gauge while its embedding cosine to the
-  teacher is **−0.02** — orthogonal, not absent. One orthogonal Procrustes
-  rotation fitted on 2,500 held-in pairs recovers agreement to **0.50** and
-  zero-shot from 0.109 to **0.240** — replicated on both seeds with
-  post-rotation agreements matching to 0.004 (`ledgers/frame_check.json`,
-  `ledgers/frame_check_s1.json`). The 5-teacher consensus arm tells the same
-  story: rotated agreement
-  **0.713 equals direct mimicry's 0.714**. Losses without a cross-space
-  anchor learn the geometry *with the global frame free* — evaluate them
-  through a fitted rotation or they read as false floors.
-- **Coupling wins retrieval.** Batch-coupled InfoNCE beats the decoupled
-  SigLIP-style pairwise loss on retrieval 2/2 at batch 256 (single batch
-  size tested — disclosed confound).
-- **Geometric add-on terms were inert** here: InfoNCE alone already lands the
-  embedding-spread band the extra terms target.
 ## Usage
@@ -77,72 +66,68 @@ import torch
 from huggingface_hub import hf_hub_download
 repo = "AbstractPhil/clip-vitb-mini-distilled"
-ckpt = hf_hub_download(repo, "student_infonce_s0.pt")
-ldr = hf_hub_download(repo, "loader.py")
 import importlib.util
 spec = importlib.util.spec_from_file_location("loader", ldr)
 loader = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(loader)
-model = loader.load_model(ckpt)              # eval mode, cuda if available
-x01 = torch.rand(4, 3, 224, 224)             # any [0,1] images
-emb = loader.embed_images(model, x01)        # (4, 512) L2-normalized
 ```
-Outputs are compatible with the teacher's text tower
-([laion/CLIP-ViT-B-16-laion2B-s34B-b88K](https://huggingface.co/laion/CLIP-ViT-B-16-laion2B-s34B-b88K))
-for zero-shot classification and retrieval.
-## Training recipe
-- Student: ViT d=240, depth 12, heads 4, patch 16, input 160px, CLS readout,
-  linear head to the teacher's 512-d projection space.
-- Targets: precomputed pooled projection-space features from
-  [AbstractPhil/bulk-coco-features](https://huggingface.co/datasets/AbstractPhil/bulk-coco-features)
-  (L2-normalized at load), teacher preprocessing verified to cosine 1.00000
-  against the live tower before any training.
-- Objective (ship arm): symmetric InfoNCE student↔teacher, temperature 0.07,
-  in-batch negatives, batch 256.
-- Optimizer: pure Adam (no weight decay), fp32, TF32 off, fixed CRC-derived
-  seeds; 8000 steps ≈ 17.3 epochs; peak 4.83 GB on one consumer GPU
-  (~37 min per run).
-- Full per-run ledgers in `ledgers/` (JSONL; `agree_cos` is cosine to the
-  teacher target; `eval_only` rows are deterministic re-evaluations, which
-  reproduce the training-run numbers to 4 decimal places).
 ## Caveats
-- COCO-only data budget. The viability bar preregistered for this experiment
-  (≥40% of teacher zero-shot at ≤10% params) was **refuted** — treat this as
-  a measured data point on the distillation-data curve, not a usable
-  general-purpose encoder.
-- Zero-shot numbers come from 160px students judging 32px CIFAR upscales;
-  absolute values are harness-scoped (floor/ceiling in the same harness
-  bracket them).
-- Single training configuration per arm (2 seeds); batch-size sensitivity of
-  the contrastive comparisons untested.
 ## Related
-- The loss-objective survey this experiment extends:
   [AbstractPhil/loss-manifest](https://huggingface.co/AbstractPhil/loss-manifest)
-- Teacher feature bank:
-  [AbstractPhil/bulk-coco-features](https://huggingface.co/datasets/AbstractPhil/bulk-coco-features)
 ## References
-- Radford et al., *Learning Transferable Visual Models From Natural Language
-  Supervision* (CLIP) — https://arxiv.org/abs/2103.00020
-- van den Oord et al., *Representation Learning with Contrastive Predictive
-  Coding* (InfoNCE) — https://arxiv.org/abs/1807.03748
 - Yang et al., *CLIP-KD: An Empirical Study of CLIP Model Distillation* —
   https://arxiv.org/abs/2307.12732
-- Wu et al., *TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight
-  Inheritance* — https://arxiv.org/abs/2309.12314
-- Zhai et al., *Sigmoid Loss for Language Image Pre-Training* (SigLIP) —
-  https://arxiv.org/abs/2303.15343
-- Lin et al., *Microsoft COCO: Common Objects in Context* —
-  https://arxiv.org/abs/1405.0312
 - Cherti et al., *Reproducible scaling laws for contrastive language-image
-  learning* (OpenCLIP / LAION-2B) — https://arxiv.org/abs/2212.07143

 tags:
 - clip
 - knowledge-distillation
+- consensus-distillation
 - infonce
 - vit
 - research
 # clip-vitb-mini-distilled
+An **8.66M-parameter ViT-Ti-class student** (~10% of a CLIP-B/16 image
+tower) distilled for CLIP-style zero-shot and retrieval — now at its
+**CC12M generation**: trained on 10.97M images against a **five-teacher
+consensus target**, shipped with the small frozen rotation that places it
+in the deployment frame.
+**Headline (CIFAR-10 / CIFAR-100 zero-shot, COCO R@1, both seeds):**
+| model | zs C10 | zs C100 | R@1 | R@5 |
 |---|---|---|---|---|
+| teacher CLIP-B/16 LAION-2B (86M) | .946 | .759 | .417 | .669 |
+| **champion: consensus + rotation (s1)** | **.5708** | **.2487** | **.1360** | **.3300** |
+| champion seed 0 | .5412 | .2210 | .1322 | .3140 |
+| best single-teacher student (InfoNCE, s0) | .5208 | .1958 | .1252 | .2968 |
+| previous release (COCO-budget InfoNCE) | .2668 | .0939 | .0580 | .1664 |
+The champion reaches **~48% of the teacher's zero-shot capability at 10%
+of its parameters** — up from 21% one data-generation earlier, with the
+data-scaling curve still unbent.
+## The finding this model embodies
+**Don't distill against your serving teacher — distill against a
+consensus of teachers, and ship a rotation.** The champion was trained
+with a composite objective (symmetric InfoNCE + per-sample MSE) against
+the *generalized Procrustes mean* of five CLIP towers (two B/16, three
+B/32; LAION-2B, OpenAI, DataComp-XL pretrainings), not against the
+deployment teacher. Its raw outputs live in the consensus frame; one
+**frozen 512×512 orthogonal rotation** (0.5MB, fitted offline on 2,500
+COCO-val pairs) maps them into the CLIP-B/16-LAION-2B frame — where it
+**beats the student trained directly against that teacher on every task
+gauge, both seeds**, at identical training cost.
+Two related phenomena, measured across the full 9-objective × 2-seed
+matrix and shipped in the ledgers:
+- **Frame ambiguity**: consensus/relational objectives learn geometry
+  with the global frame unpinned; absolute gauges misread them as
+  failures until a fitted rotation is applied. Absolute-target consensus
+  retains *all* its geometry (rotation recovers everything and more);
+  purely relational similarity-matching degrades absolutely at scale.
+- **The anchor dial**: across recipes, anchor (MSE) weight trades
+  teacher-agreement against retrieval monotonically; the composite sits
+  at the sweet point, and its InfoNCE term is what buys retrieval
+  (pure-MSE consensus, even rotated, reaches only R@1 ≈ .05).
 ## Usage
 from huggingface_hub import hf_hub_download
 repo = "AbstractPhil/clip-vitb-mini-distilled"
+ckpt = hf_hub_download(repo, "cc12m/consensus_nce_mse_s1_t88000.pt")
+rot  = hf_hub_download(repo, "cc12m/rotation_s1.pt")
+ldr  = hf_hub_download(repo, "loader.py")
 import importlib.util
 spec = importlib.util.spec_from_file_location("loader", ldr)
 loader = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(loader)
+model = loader.load_model(ckpt)
+R = loader.load_rotation(rot)
+x01 = torch.rand(4, 3, 224, 224)                  # any [0,1] images
+emb = loader.embed_images(model, x01, rotation=R)  # (4, 512), LAION frame
 ```
+Outputs (with the rotation) are compatible with the
+[LAION B/16 text tower](https://huggingface.co/laion/CLIP-ViT-B-16-laion2B-s34B-b88K)
+for zero-shot and retrieval. The single-teacher alternate
+(`cc12m/infonce_s0_t88000.pt`) needs no rotation. The original
+COCO-budget model (`student_infonce_s0.pt`) remains for provenance.
+## Training
+- Data: CC12M (10,968,539 images), 88,000 steps at batch 256 ≈ 2.05
+  epochs; precomputed teacher targets from
+  [bulk-cc12m-features](https://huggingface.co/datasets/AbstractPhil/bulk-cc12m-features)
+  (zero teacher inference during training).
+- Champion objective: symmetric InfoNCE (t=0.07) + per-sample MSE against
+  the five-tower GPA consensus; pure Adam 3e-4, no weight decay, fp32.
+- Every number above is deterministic: evaluation re-runs reproduce to 4
+  decimal places, and the shipped loader reproduces the training-harness
+  numbers exactly (verified cross-machine before this release).
 ## Caveats
+- Research miniature: ~48% of teacher zero-shot, not a production
+  encoder. The scaling trend (21% → 42% → 48% across data generations)
+  is the point.
+- CIFAR evals upscale 32px images through the student's 160px input;
+  floor/ceiling brackets in the same harness contextualize them.
+- Two seeds per objective; single batch size (256).
 ## Related
+- Teacher feature banks (two towers, viewer-ready):
+  [AbstractPhil/bulk-cc12m-features](https://huggingface.co/datasets/AbstractPhil/bulk-cc12m-features)
+  · COCO companion: [AbstractPhil/bulk-coco-features](https://huggingface.co/datasets/AbstractPhil/bulk-coco-features)
+- The loss-objective genealogy this line extends:
   [AbstractPhil/loss-manifest](https://huggingface.co/AbstractPhil/loss-manifest)
+- Consensus encoder line (COCO scale):
+  [AbstractPhil/geolip-vit-captionbank-coco](https://huggingface.co/AbstractPhil/geolip-vit-captionbank-coco)
 ## References
+- Radford et al., *Learning Transferable Visual Models From Natural
+  Language Supervision* (CLIP) — https://arxiv.org/abs/2103.00020
+- van den Oord et al., *Representation Learning with Contrastive
+  Predictive Coding* (InfoNCE) — https://arxiv.org/abs/1807.03748
 - Yang et al., *CLIP-KD: An Empirical Study of CLIP Model Distillation* —
   https://arxiv.org/abs/2307.12732
+- Gower, *Generalized Procrustes Analysis* —
+  https://doi.org/10.1007/BF02291478
+- Changpinyo et al., *Conceptual 12M* — https://arxiv.org/abs/2102.08981
 - Cherti et al., *Reproducible scaling laws for contrastive language-image
+  learning* — https://arxiv.org/abs/2212.07143