adipanda commited on
Commit
db899ff
·
verified ·
1 Parent(s): a5eba26

final write-up

Browse files
Files changed (1) hide show
  1. README.md +31 -0
README.md ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # TC-LeJEPA (Text-Conditioned LeJEPA)
2
+
3
+ An ablation of adding text conditioning to the predictor in [LeJEPA](https://arxiv.org/abs/2511.08544). We do **not** predict text — we condition the JEPA predictor on text and keep the original two-term objective `L = (1-λ)·L_inv + λ·SIGReg`.
4
+
5
+ ## Variants
6
+ | Variant | Conditioning |
7
+ |-------------|-----------------------------------------|
8
+ | `baseline` | vanilla LeJEPA, MLP predictor |
9
+ | `film` | FiLM on MLP predictor, text → (γ, β) |
10
+ | `xattn` | Patch tokens cross-attend to text |
11
+ | `wrong_text`| `xattn` with permuted label-text map |
12
+
13
+ Backbone: ViT-Small/16 at 128×128. Text tower: OpenCLIP ViT-B/32 (frozen). Dataset: CIFAR-100.
14
+
15
+ ## Reading order
16
+ 1. `comparison.md` — results, figures, and answers to the four research questions.
17
+ 2. `tclejepa_src/modules.py` — `TCLeJEPAModel`, predictor variants, `SIGReg`.
18
+ 3. `tclejepa_src/train.py` — training loop (same loss for every variant).
19
+ 4. `tclejepa_src/evaluate.py` — linear probe, SIGReg↔acc correlation, t-SNE steering.
20
+
21
+ ## Artifacts
22
+ Checkpoints, figures, logs, and `comparison.{md,json}` live at
23
+ **https://huggingface.co/adipanda/lejepa**.
24
+
25
+ ## Running
26
+ ```bash
27
+ set -a && source .env && set +a # loads WANDB_API_KEY and HF_TOKEN
28
+ uv sync
29
+ EPOCHS=30 BS=512 WORKERS=12 ./run_all.sh # runs all 4 variants sequentially
30
+ uv run python -m tclejepa_src.evaluate # produces comparison.{json,md} + figures
31
+ ```