---
language: en
license: apache-2.0
tags:
- self-supervised-learning
- knowledge-distillation
- vision-transformer
- model-compression
---

# TinySSL: Distilling Foundation Model Features for Resource-Efficient Vision

**Authors**: Emran Abdu  
**DOI**: [10.5281/zenodo.21180996](https://zenodo.org/record/21180996)  
**Code**: [GitHub](https://github.com/Emran-goat/tinyssl)  
**License**: Apache 2.0

## Abstract

Vision foundation models like DINOv2 produce powerful representations, but training them costs millions of dollars in GPU compute. We introduce TinySSL, a 2.8M-parameter framework that distills frozen DINOv2-S/14 features into a compact CNN-transformer hybrid. A composite loss combines masked image modeling with JEPA alignment, cosine feature matching, and KoLeo uniformity regularization, removing the need for negative pairs, momentum encoders, or large batches. A progressive augmentation curriculum stabilizes training on commodity hardware. Across four domain benchmarks (Flowers102, Oxford Pets, EuroSAT, BreastMNIST), TinySSL retains over 97% of DINOv2-S/14 linear-probe accuracy with a 7x parameter reduction and trains in under 30 minutes on a single CPU.

## Citation

`ibtex
@article{abdu2026tinyssl,
  title={TinySSL: Distilling Foundation Model Features for Resource-Efficient Vision},
  author={Emran Abdu},
  year={2026},
  doi={10.5281/zenodo.21180996}
}
`