---
library_name: pytorch
tags:
  - self-supervised-learning
  - vision-transformer
  - jepa
  - representation-learning
  - pytorch
datasets:
  - stl10
  - cifar10
pipeline_tag: image-feature-extraction
---

# Self-Supervised to JEPA — pretrained ViT-Tiny checkpoints

Pre-training checkpoints for **[self-supervised-to-jepa](https://github.com/hugoplanchat/SNU-Internship)**: a controlled benchmark of ten self-supervised methods re-implemented from scratch on a **single shared ViT-Tiny** (patch 4, d=128, 6 layers, 4 heads, ≈1.2M params), pre-trained on STL-10 (unlabeled, 100 epochs) and evaluated on CIFAR-10. These are small research models meant to reproduce this benchmark, not general-purpose backbones — useful to regenerate every figure and downstream number without re-training on a GPU.

## What is in this repo (for now)

The **10 pre-training checkpoints**, one per method. Each `*_pretrain.pt` holds, besides the SSL weights, the full monitoring history recorded during training:

| key | content |
|---|---|
| `model` | full SSL model `state_dict` (all branches) |
| `encoder` | the shared ViT-Tiny trunk `state_dict` (the reusable part) |
| `losses` (or `hist["loss"]`) | per-epoch training loss |
| `knn_history` | list of (epoch, weighted k-NN accuracy) pairs on CIFAR-10 |
| `rank_history` | list of (epoch, effective rank) pairs (MAE stores `lr_history` instead) |
| method-specific loss terms | the objective's component breakdown: SimCLR `pos`/`neg`, MoCo `lookup`, BYOL feature `std`, CPC `row_acc`, VICReg `inv`/`var`/`cov`, LeJEPA `pred`/`sig`; DINOv3 and VISReg bundle theirs in a `hist` dict |

Linear-probe and fine-tune checkpoints will be added later.

## Downstream results (CIFAR-10, %)

| Method | k-NN | Linear probe | Fine-tune |
|---|---|---|---|
| MAE | 46.77 | 49.05 | 83.57 |
| CPC | 39.93 | 36.50 | 73.67 |
| SimCLR | **74.17** | 72.92 | **86.75** |
| MoCo | 74.13 | 69.68 | 84.95 |
| BYOL | 73.07 | **73.76** | 86.23 |
| DINOv3 | 59.13 | 54.98 | 81.90 |
| VICReg | 73.04 | 73.08 | 86.65 |
| I-JEPA | 57.71 | 56.27 | 83.25 |
| LeJEPA | 65.06 | 62.00 | 81.83 |
| VISReg | 71.57 | 68.87 | 85.48 |

Supervised ViT-Tiny (from scratch, reference): 78.69 fine-tune.

## Files

Checkpoints mirror the repo layout, so you can drop them straight into `ssl/checkpoints/`:

```
04_mae/mae_pretrain.pt        09_dinov3/dinov3_pretrain.pt
05_cpc/cpc_pretrain.pt        10_VICReg/vicreg_pretrain.pt
06_simclr/simclr_pretrain.pt  11_IJEPA/ijepa_pretrain.pt
07_moco/moco_pretrain.pt      12_LEJEPA/lejepa_pretrain.pt
08_byol/byol_pretrain.pt      13_VISReg/visreg_pretrain.pt
```

## Load a checkpoint

The `.pt` files are `state_dict`s: you need the model classes from the [GitHub repo](https://github.com/hugoplanchat/SNU-Internship) to instantiate the architecture, then pour the weights in.

```python
import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("HugoPlanchat/self-supervised-to-jepa", "08_byol/byol_pretrain.pt")
ck = torch.load(path, map_location="cpu", weights_only=False)

# ck["encoder"] is the shared ViT-Tiny trunk; rebuild ViTEncoder() from the notebook, then:
# encoder.load_state_dict(ck["encoder"])

print(ck.keys())                 # weights + monitoring histories
print(ck["knn_history"][-1])     # e.g. (100, 0.7307)
```

To make the notebooks *restart-and-run-all instant*, download each file into `ssl/checkpoints/<method>/` and the load-or-train guard will reload instead of training.

## Links

Code, report and full write-up: https://github.com/hugoplanchat/SNU-Internship