Contrastive Pocket
Contrastive Pocket pretrains a compact vision encoder without labels. Two independently augmented views of every 8x8 digit are pulled together while other examples in the batch act as negatives.
After pretraining, the encoder is frozen. Logistic-regression probes measure held-out classification at 10, 25, and all labeled examples per class. Architecture-matched random encoders and raw-pixel probes provide controls.
Reproduce
uv run python projects/tiny-vision-foundry/prepare_data.py
uv run python projects/contrastive-pocket/train.py
Digit labels are never used in the contrastive optimizer.
Verified results
| Labels per class | Contrastive encoder | Random encoder | Raw pixels |
|---|---|---|---|
| 10 | 89.26% | 79.26% | 90.74% |
| 25 | 92.96% | 82.59% | 93.70% |
| 100 | 96.30% | 81.11% | 97.41% |
The 29,984-parameter encoder substantially outperformed architecture-matched random features at every label budget, while raw pixels remained slightly stronger on this small 8x8 corpus. Its augmented-positive versus mismatched-pair cosine gap was 0.3435, compared with 0.0150 for the random encoder, and its mean embedding standard deviation was 0.1537, providing an explicit non-collapse check.