| language: en | |
| license: apache-2.0 | |
| tags: | |
| - self-supervised-learning | |
| - knowledge-distillation | |
| - vision-transformer | |
| - model-compression | |
| # TinySSL: Distilling Foundation Model Features for Resource-Efficient Vision | |
| **Authors**: Emran Abdu | |
| **DOI**: [10.5281/zenodo.21180996](https://zenodo.org/record/21180996) | |
| **Code**: [GitHub](https://github.com/Emran-goat/tinyssl) | |
| **License**: Apache 2.0 | |
| ## Abstract | |
| Vision foundation models like DINOv2 produce powerful representations, but training them costs millions of dollars in GPU compute. We introduce TinySSL, a 2.8M-parameter framework that distills frozen DINOv2-S/14 features into a compact CNN-transformer hybrid. A composite loss combines masked image modeling with JEPA alignment, cosine feature matching, and KoLeo uniformity regularization, removing the need for negative pairs, momentum encoders, or large batches. A progressive augmentation curriculum stabilizes training on commodity hardware. Across four domain benchmarks (Flowers102, Oxford Pets, EuroSAT, BreastMNIST), TinySSL retains over 97% of DINOv2-S/14 linear-probe accuracy with a 7x parameter reduction and trains in under 30 minutes on a single CPU. | |
| ## Citation | |
| `ibtex | |
| @article{abdu2026tinyssl, | |
| title={TinySSL: Distilling Foundation Model Features for Resource-Efficient Vision}, | |
| author={Emran Abdu}, | |
| year={2026}, | |
| doi={10.5281/zenodo.21180996} | |
| } | |
| ` | |