final write-up

comparison.md

@@ -0,0 +1,87 @@
+# TC-LeJEPA ablation — results
+
+Text-conditioned LeJEPA: baseline vs two conditioning architectures (FiLM, cross-attention) vs a wrong-label control. Backbone = ViT-Small/16 @ 128×128. Text tower = OpenCLIP ViT-B/32 (frozen). Dataset = CIFAR-100. Loss = `(1-λ)·L_inv + λ·SIGReg`, λ = 0.02. Artifacts: https://huggingface.co/adipanda/lejepa
+
+## Ablation table
+
+| Variant | Linear probe top-1 | SIGReg↔acc Spearman | Loss smoothness (last 50%) | Notes |
+|---|---|---|---|---|
+| `baseline` | 0.3502 | -0.943 (n=6) | 0.00270 | vanilla LeJEPA, no text path. |
+| `film` | 0.2941 | -1.000 (n=6) | 0.00120 | MLP predictor with FiLM(text) on each hidden layer; γ,β init=(1,0). |
+| `xattn` | 0.2313 | -0.943 (n=6) | 0.00282 | Patch tokens cross-attend to text; zero-init output proj. |
+| `wrong_text` | 0.2303 | -0.886 (n=6) | 0.00175 | xattn architecture with permuted label-text map — probes whether text **content** matters. |
+
+## Per-variant findings
+
+- **`baseline`** — best probe top-1 **0.3502** at epoch 29; final train `lejepa`=0.2282, final online probe=0.2496.
+- **`film`** — best probe top-1 **0.2941** at epoch 29; final train `lejepa`=0.0850, final online probe=0.2028.
+- **`xattn`** — best probe top-1 **0.2313** at epoch 29; final train `lejepa`=0.1334, final online probe=0.1404.
+- **`wrong_text`** — best probe top-1 **0.2303** at epoch 24; final train `lejepa`=0.1357, final online probe=0.1456.
+
+## Research questions — direct answers
+
+1. **Does text conditioning beat vanilla LeJEPA at matched epochs and λ?** — **No**, baseline at 0.3502 ≥ best conditioned (film, 0.2941).
+2. **FiLM vs cross-attention — which wins and by how much?** — **FiLM wins**: 0.2941 vs xattn 0.2313 (Δ = +6.28 pp).
+3. **Does text content matter (correct vs wrong vs none)?** — **No (regularization-only effect)**: wrong_text 0.2303 ≈ xattn 0.2313 (Δ = -0.10 pp). Text **content** does not matter at this scale — conditioning is acting as a regularizer, not a semantic signal.
+   - baseline (no text): 0.3502   xattn: 0.2313   wrong_text: 0.2303
+4. **Does the SIGReg↔accuracy correlation survive under conditioning?**
+   - `baseline`: ρ = -0.943 (n=6)
+   - `film`: ρ = -1.000 (n=6)
+   - `xattn`: ρ = -0.943 (n=6)
+   - `wrong_text`: ρ = -0.886 (n=6)
+   - A strong negative ρ is the *expected* sign for LeJEPA: lower SIGReg ⇒ closer to iso-Gaussian in projection space ⇒ higher linear probe. Conditioning preserves this whenever |ρ| stays close to baseline's.
+
+## Figures
+
+![Training loss and online probe accuracy per epoch, per variant.](figures/loss_curves.png)
+*Training loss and online probe accuracy per epoch, per variant.*
+
+![SIGReg(val) vs linear probe — baseline.](figures/sigreg_vs_acc_baseline.png)
+*SIGReg(val) vs linear probe — baseline.*
+
+![SIGReg(val) vs linear probe — FiLM.](figures/sigreg_vs_acc_film.png)
+*SIGReg(val) vs linear probe — FiLM.*
+
+![SIGReg(val) vs linear probe — cross-attention.](figures/sigreg_vs_acc_xattn.png)
+*SIGReg(val) vs linear probe — cross-attention.*
+
+![SIGReg(val) vs linear probe — wrong-label text.](figures/sigreg_vs_acc_wrong_text.png)
+*SIGReg(val) vs linear probe — wrong-label text.*
+
+![t-SNE of projections under three conditioning prompts (FiLM).](figures/tsne_steer_film.png)
+*t-SNE of projections under three conditioning prompts (FiLM).*
+
+![t-SNE of projections under three conditioning prompts (xattn).](figures/tsne_steer_xattn.png)
+*t-SNE of projections under three conditioning prompts (xattn).*
+
+![t-SNE of projections under three conditioning prompts (wrong_text).](figures/tsne_steer_wrong_text.png)
+*t-SNE of projections under three conditioning prompts (wrong_text).*
+
+
+## Setup parity checklist
+
+- **Backbone**: ViT-Small/16 @ 128×128, `drop_path_rate=0.1`, no pretraining — identical across variants.
+- **Data**: CIFAR-100 train (50k) / test (10k); shared split, shared two-view augmentation.
+- **Optimizer**: AdamW, `lr=1e-3`, `wd=5e-2`, grad clip 1.0, bf16 autocast, cosine schedule + 2-epoch warmup.
+- **λ**: 0.02 for all variants.
+- **Epochs & batch**: 30 epochs × 512 batch × 2 views, identical.
+- **Eval harness**: same feature extractor, same StandardScaler + logistic regression probe on train/val, same SIGReg probe on ≤4k val projections.
+- **Text tower**: OpenCLIP ViT-B/32 (frozen), `.eval()`, no gradient. Only the text-projection layer inside the predictor is trainable on the text path.
+
+## Honest caveats
+
+- **Single dataset** (CIFAR-100, 128×128 upsampled). Findings may not transfer to ImageNet-scale pretraining.
+- **Frozen text encoder**: we use CLIP ViT-B/32 text features verbatim (only the predictor's text-projection layer adapts).
+- **Ablation-scale**: 30 epochs is short for SSL; absolute numbers are lower than published LeJEPA results.
+- **Linear probe only**: no fine-tuning, no transfer to downstream tasks, no k-NN or MLP probe.
+- **Class-name prompts only**: `"a photo of a <class>"`. Richer prompts (scenes, attributes) are left for future work.
+- **SIGReg on val** uses single-view features rather than the two-view training population; it is still comparable across variants (same harness).
+
+## Suggested next experiments
+
+- Longer schedules (200+ epochs) and larger backbones (ViT-B/16).
+- Richer text prompts (captions, attributes) rather than class names alone.
+- Downstream transfer (CIFAR-10, STL-10, ImageNet-100 linear probe and full fine-tuning).
+- **Unfreeze** the text tower (or its last block) — does adaptation improve the conditioning signal?
+- Temporal extension à la **LeWorldModel**: condition the predictor on future-frame captions.
+- Per-patch prediction target (true JEPA-style masked prediction) rather than global projection, to match the instruction's 'target patch embeddings' framing more literally.