Overcomplete Sparse Autoencoders for Visual Representation Learning (STL-10)

Controlled study of whether an overcomplete, L1-sparse convolutional autoencoder learns features that transfer better to STL-10 classification than a conventional undercomplete autoencoder, against a supervised ResNet-18 trained from scratch, at 10 / 25 / 50 / 100 % of the labelled data.

Code: https://github.com/0x-khalil/OSAE

  • Shared feature extractor: ResNet-18 adapted to 96×96 (3×3/2 stem, no max-pool); identical downstream classifier for all models.
  • Standard AE latent: 128×6×6 = 4,608 coefficients (0.17× input). Overcomplete AE latent: 2048×6×6 = 73,728 (2.67× input).
  • Pre-training on the 100,000 unlabeled images; λ* = 0.01 (selected on validation).
  • Downstream: full fine-tuning and frozen-encoder protocols, 3 seeds, identical stratified subsets, validation-only model selection, test split used once for final evaluation.

Files

Path Content
results/RESULTS.md headline table, ablations, paired statistics, costs
results/figures/ label-efficiency curves, confusion matrices, per-class accuracy
results/representation/ sparsity, reconstruction, separability, PCA / t-SNE / UMAP, CKA
encoders/<variant>/encoder.pt pre-trained ResNet-18 trunk (torch.load(...)["encoder"])
downstream/ per-run metrics (and checkpoints if uploaded)
configs/, data_splits/ exact configuration and split indices

Load a pre-trained trunk with the code from the repository:

import torch
from osae.models import Classifier
clf = Classifier()
clf.encoder.load_state_dict(torch.load("encoders/oae_sparse/encoder.pt")["encoder"])

Results

All numbers: mean ± std over seeds, in %. Test = official STL-10 test split (8,000 images); every choice below was made on the validation split.

finetune — test_top1

Model variant 10% 25% 50% 100%
ResNet-18 baseline scratch 44.11 ± 0.62 54.44 ± 1.86 65.16 ± 0.97 72.77 ± 0.40
Standard AE C128_l0_clean 49.19 ± 0.28 58.44 ± 0.29 65.80 ± 0.12 68.91 ± 0.64
Sparse OAE C2048_l0.01_mask0.5 49.55 ± 0.84 59.21 ± 0.54 67.40 ± 0.14 69.82 ± 4.37

finetune — test_macro_f1

Model variant 10% 25% 50% 100%
ResNet-18 baseline scratch 44.00 ± 0.65 54.40 ± 1.93 65.21 ± 0.97 72.87 ± 0.39
Standard AE C128_l0_clean 49.02 ± 0.28 58.46 ± 0.28 65.84 ± 0.16 68.81 ± 0.77
Sparse OAE C2048_l0.01_mask0.5 49.36 ± 0.82 59.16 ± 0.46 67.45 ± 0.15 69.76 ± 4.46

finetune — val_top1

Model variant 10% 25% 50% 100%
ResNet-18 baseline scratch 47.47 ± 2.31 57.20 ± 1.22 64.40 ± 1.59 70.33 ± 1.10
Standard AE C128_l0_clean 51.13 ± 0.42 57.07 ± 0.95 62.27 ± 1.50 64.33 ± 1.30
Sparse OAE C2048_l0.01_mask0.5 50.40 ± 1.83 59.33 ± 0.31 64.07 ± 0.70 66.80 ± 0.53

finetune — all variants (test_top1)

variant latent lambda noise 10% 25% 50% 100% val (all f)
ae_sparse 128 0.01 none 47.90 ± 0.87 57.55 ± 0.64 64.12 ± 0.63 67.90 ± 0.79 59.02
ae_sparse_denoise 128 0.01 gaussian 48.93 ± 0.15 57.14 ± 0.50 63.78 ± 0.94 68.45 ± 0.51 59
ae 128 0.0 none 49.19 ± 0.28 58.44 ± 0.29 65.80 ± 0.12 68.91 ± 0.64 58.7
ae_denoise 128 0.0 gaussian 48.88 ± 0.48 58.58 ± 0.24 65.35 ± 0.73 69.50 ± 0.35 59.17
oae_l0.01/oae_sparse 2048 0.01 none 49.22 ± 0.51 57.58 ± 0.64 64.71 ± 0.39 67.90 ± 0.99 57.92
oae_sparse_denoise 2048 0.01 gaussian 48.85 ± 0.66 57.60 ± 0.53 65.05 ± 0.52 69.51 ± 0.14 58.5
oae_sparse_mask 2048 0.01 mask 49.55 ± 0.84 59.21 ± 0.54 67.40 ± 0.14 69.82 ± 4.37 60.15
oae 2048 0.0 none 49.77 ± 0.46 55.69 ± 1.58 65.09 ± 0.33 69.55 ± 0.48 58.33
oae_denoise 2048 0.0 gaussian 48.93 ± 0.40 57.85 ± 0.46 65.55 ± 0.16 69.77 ± 0.21 59.35
scratch – – – 44.11 ± 0.62 54.44 ± 1.86 65.16 ± 0.97 72.77 ± 0.40 59.85

finetune — downstream training cost (mean over runs)

canonical train_time_s peak_train_mem_mb params_total params_trainable
C128_l0.01_clean 147.5 919.2 1.1174e+07 1.1174e+07
C128_l0.01_gauss0.5 145.1 919.2 1.1174e+07 1.1174e+07
C128_l0_clean 153 919.2 1.1174e+07 1.1174e+07
C128_l0_gauss0.5 149.9 919.2 1.1174e+07 1.1174e+07
C2048_l0.01_clean 148.4 919.2 1.1174e+07 1.1174e+07
C2048_l0.01_gauss0.5 142 919.2 1.1174e+07 1.1174e+07
C2048_l0.01_mask0.5 143.9 919.2 1.1174e+07 1.1174e+07
C2048_l0_clean 148.7 919.2 1.1174e+07 1.1174e+07
C2048_l0_gauss0.5 147.4 919.2 1.1174e+07 1.1174e+07
scratch 151 919.2 1.1174e+07 1.1174e+07

finetune — paired comparisons (test top-1, B − A in points)

A B fraction n_seeds diff_B_minus_A diff_std p_paired_t p_paired_t_holm cohens_dz p_mcnemar_max_over_seeds
scratch C128_l0_clean 0.1 3 5.083 0.719 0.006602 0.02641 7.07 2.226e-14
scratch C128_l0_clean 0.25 3 4.004 2.073 0.07893 0.1579 1.931 0.002828
scratch C128_l0_clean 0.5 3 0.6417 1.079 0.4112 0.4112 0.5947 0.3509
scratch C128_l0_clean 1 3 -3.858 0.5837 0.007542 0.02641 -6.61 7.597e-11
scratch C2048_l0.01_mask0.5 0.1 3 5.438 0.9135 0.009278 0.03711 5.952 6.667e-17
scratch C2048_l0.01_mask0.5 0.25 3 4.775 1.338 0.0252 0.07559 3.568 2.118e-09
scratch C2048_l0.01_mask0.5 0.5 3 2.237 1.038 0.06488 0.1298 2.155 0.01869
scratch C2048_l0.01_mask0.5 1 3 -2.95 4.179 0.346 0.346 -0.7059 0.5993
C128_l0_clean C2048_l0.01_mask0.5 0.1 3 0.3542 1.122 0.6394 1 0.3157 0.3909
C128_l0_clean C2048_l0.01_mask0.5 0.25 3 0.7708 0.8079 0.2402 0.7207 0.9541 0.7619
C128_l0_clean C2048_l0.01_mask0.5 0.5 3 1.596 0.1938 0.004879 0.01952 8.235 0.004674
C128_l0_clean C2048_l0.01_mask0.5 1 3 0.9083 3.722 0.7136 1 0.244 4.28e-09
C128_l0_clean C2048_l0_clean 0.1 3 0.5792 0.6411 0.2581 0.3788 0.9034 0.8093
C128_l0_clean C2048_l0_clean 0.25 3 -2.75 1.875 0.1263 0.3788 -1.467 0.05449
C128_l0_clean C2048_l0_clean 0.5 3 -0.7125 0.219 0.03008 0.1203 -3.253 0.2517
C128_l0_clean C2048_l0_clean 1 3 0.6375 0.553 0.1839 0.3788 1.153 1
C2048_l0_clean C2048_l0.01_clean 0.1 3 -0.55 0.2382 0.05719 0.2288 -2.309 0.3679
C2048_l0_clean C2048_l0.01_clean 0.25 3 1.887 1.003 0.08257 0.2477 1.883 0.01547
C2048_l0_clean C2048_l0.01_clean 0.5 3 -0.3792 0.7142 0.4549 0.4549 -0.5309 0.3409
C2048_l0_clean C2048_l0.01_clean 1 3 -1.646 1.06 0.1149 0.2477 -1.553 0.1841
C128_l0_clean C128_l0.01_clean 0.1 3 -1.288 0.7253 0.0915 0.2745 -1.775 0.1862
C128_l0_clean C128_l0.01_clean 0.25 3 -0.8917 0.7381 0.1715 0.343 -1.208 0.7835
C128_l0_clean C128_l0.01_clean 0.5 3 -1.683 0.5693 0.03608 0.1443 -2.957 0.01671
C128_l0_clean C128_l0.01_clean 1 3 -1.008 1.264 0.3012 0.343 -0.7977 0.4219
C2048_l0.01_clean C2048_l0.01_gauss0.5 0.1 3 -0.3708 0.4537 0.2925 0.8775 -0.8174 0.824
C2048_l0.01_clean C2048_l0.01_gauss0.5 0.25 3 0.01667 1.096 0.9814 0.9814 0.01521 0.2753
C2048_l0.01_clean C2048_l0.01_gauss0.5 0.5 3 0.3417 0.6825 0.4773 0.9546 0.5006 0.6171
C2048_l0.01_clean C2048_l0.01_gauss0.5 1 3 1.613 0.8928 0.08878 0.3551 1.806 0.02627
C128_l0_clean C128_l0_gauss0.5 0.1 3 -0.3167 0.3224 0.231 0.9241 -0.9821 0.9317
C128_l0_clean C128_l0_gauss0.5 0.25 3 0.1417 0.4161 0.6152 0.9852 0.3404 0.4568
C128_l0_clean C128_l0_gauss0.5 0.5 3 -0.4583 0.6193 0.3284 0.9852 -0.7401 0.6854
C128_l0_clean C128_l0_gauss0.5 1 3 0.5917 0.9863 0.4079 0.9852 0.5999 0.5875

frozen — test_top1

Model variant 10% 25% 50% 100%
Random frozen ResNet-18 (control) random 35.58 ± 0.28 39.48 ± 0.34 41.61 ± 0.72 42.82 ± 1.13
Standard AE C128_l0_clean 43.90 ± 0.29 48.54 ± 1.39 50.90 ± 0.56 52.55 ± 0.21
Sparse OAE C2048_l0.01_mask0.5 45.72 ± 0.18 49.67 ± 1.44 53.93 ± 0.09 55.66 ± 0.30

frozen — test_macro_f1

Model variant 10% 25% 50% 100%
Random frozen ResNet-18 (control) random 35.26 ± 0.24 39.22 ± 0.38 41.17 ± 0.66 42.36 ± 1.14
Standard AE C128_l0_clean 43.83 ± 0.30 48.31 ± 1.34 50.68 ± 0.62 52.26 ± 0.26
Sparse OAE C2048_l0.01_mask0.5 45.50 ± 0.19 49.52 ± 1.48 53.86 ± 0.08 55.54 ± 0.33

frozen — val_top1

Model variant 10% 25% 50% 100%
Random frozen ResNet-18 (control) random 37.20 ± 1.56 41.93 ± 1.01 42.80 ± 3.33 45.27 ± 2.34
Standard AE C128_l0_clean 45.60 ± 0.20 50.27 ± 0.81 50.73 ± 1.14 52.33 ± 1.21
Sparse OAE C2048_l0.01_mask0.5 46.33 ± 0.50 51.20 ± 0.20 54.53 ± 0.83 56.60 ± 0.53

frozen — all variants (test_top1)

variant latent lambda noise 10% 25% 50% 100% val (all f)
ae_sparse 128 0.01 none 44.01 ± 0.35 48.34 ± 0.25 50.41 ± 0.13 51.15 ± 0.36 49
ae_sparse_denoise 128 0.01 gaussian 42.26 ± 0.89 47.79 ± 0.43 49.28 ± 0.30 50.71 ± 0.29 47.5
ae 128 0.0 none 43.90 ± 0.29 48.54 ± 1.39 50.90 ± 0.56 52.55 ± 0.21 49.73
ae_denoise 128 0.0 gaussian 42.89 ± 0.56 48.73 ± 0.47 51.08 ± 0.47 52.66 ± 0.31 48.43
oae_l0.001 2048 0.001 none 44.61 – – 53.02 48.3
oae_l0.01/oae_sparse 2048 0.01 none 42.94 ± 0.62 48.20 ± 0.14 50.65 ± 0.57 52.07 ± 0.42 49.72
oae_sparse_denoise 2048 0.01 gaussian 43.00 ± 0.61 48.59 ± 1.46 51.33 ± 0.41 52.85 ± 0.26 49.53
oae_sparse_mask 2048 0.01 mask 45.72 ± 0.18 49.67 ± 1.44 53.93 ± 0.09 55.66 ± 0.30 52.17
oae_l0.1 2048 0.1 none 42.27 – – 50.56 46.5
oae 2048 0.0 none 44.22 ± 0.91 49.39 ± 0.86 52.15 ± 0.20 52.88 ± 0.85 51.03
oae_denoise 2048 0.0 gaussian 44.13 ± 0.45 50.03 ± 0.24 52.55 ± 0.27 54.14 ± 0.10 50.67
random – – – 35.58 ± 0.28 39.48 ± 0.34 41.61 ± 0.72 42.82 ± 1.13 41.8

frozen — downstream training cost (mean over runs)

canonical train_time_s peak_train_mem_mb params_total params_trainable
C128_l0.01_clean 59.8 832.6 1.1174e+07 5130
C128_l0.01_gauss0.5 57.3 832.6 1.1174e+07 5130
C128_l0_clean 56.6 832.6 1.1174e+07 5130
C128_l0_gauss0.5 59.4 832.6 1.1174e+07 5130
C2048_l0.001_clean 64.2 832.6 1.1174e+07 5130
C2048_l0.01_clean 56.5 832.6 1.1174e+07 5130
C2048_l0.01_gauss0.5 56.6 832.6 1.1174e+07 5130
C2048_l0.01_mask0.5 57.9 832.6 1.1174e+07 5130
C2048_l0.1_clean 66.5 832.6 1.1174e+07 5130
C2048_l0_clean 61 832.6 1.1174e+07 5130
C2048_l0_gauss0.5 56.5 832.6 1.1174e+07 5130
random 232.8 2418.4 1.1174e+07 5130

frozen — paired comparisons (test top-1, B − A in points)

A B fraction n_seeds diff_B_minus_A diff_std p_paired_t p_paired_t_holm cohens_dz p_mcnemar_max_over_seeds
C128_l0_clean C2048_l0.01_mask0.5 0.1 3 1.825 0.1146 0.001311 0.005244 15.93 0.0018
C128_l0_clean C2048_l0.01_mask0.5 0.25 3 1.133 2.788 0.5543 0.5543 0.4066 0.0002473
C128_l0_clean C2048_l0.01_mask0.5 0.5 3 3.029 0.5236 0.009812 0.02457 5.786 6.927e-06
C128_l0_clean C2048_l0.01_mask0.5 1 3 3.117 0.4915 0.00819 0.02457 6.341 9.169e-07
C128_l0_clean C2048_l0_clean 0.1 3 0.3208 0.657 0.4867 1 0.4884 0.8107
C128_l0_clean C2048_l0_clean 0.25 3 0.85 2.234 0.5776 1 0.3805 0.0009237
C128_l0_clean C2048_l0_clean 0.5 3 1.242 0.5646 0.06253 0.2501 2.199 0.2492
C128_l0_clean C2048_l0_clean 1 3 0.3333 0.7821 0.5372 1 0.4262 0.2995
C2048_l0_clean C2048_l0.01_clean 0.1 3 -1.279 0.8055 0.1107 0.332 -1.588 0.4022
C2048_l0_clean C2048_l0.01_clean 0.25 3 -1.192 0.9306 0.1568 0.332 -1.281 0.8221
C2048_l0_clean C2048_l0.01_clean 0.5 3 -1.496 0.6377 0.05557 0.2223 -2.346 0.09288
C2048_l0_clean C2048_l0.01_clean 1 3 -0.8042 1.172 0.3567 0.3567 -0.6861 0.3488
C128_l0_clean C128_l0.01_clean 0.1 3 0.1125 0.5717 0.7657 1 0.1968 0.5233
C128_l0_clean C128_l0.01_clean 0.25 3 -0.2 1.61 0.8496 1 -0.1242 0.2949
C128_l0_clean C128_l0.01_clean 0.5 3 -0.4958 0.6831 0.3356 1 -0.7259 1
C128_l0_clean C128_l0.01_clean 1 3 -1.396 0.2306 0.008975 0.0359 -6.053 0.01661
C2048_l0.01_clean C2048_l0.01_gauss0.5 0.1 3 0.0625 0.598 0.873 1 0.1045 0.5002
C2048_l0.01_clean C2048_l0.01_gauss0.5 0.25 3 0.3958 1.325 0.6564 1 0.2988 0.05433
C2048_l0.01_clean C2048_l0.01_gauss0.5 0.5 3 0.675 0.3356 0.07345 0.2938 2.011 0.5933
C2048_l0.01_clean C2048_l0.01_gauss0.5 1 3 0.7792 0.4661 0.1014 0.3043 1.672 0.5116
C128_l0_clean C128_l0_gauss0.5 0.1 3 -1.008 0.593 0.09853 0.3941 -1.7 0.5056
C128_l0_clean C128_l0_gauss0.5 0.25 3 0.1958 0.9284 0.7499 1 0.2109 0.203
C128_l0_clean C128_l0_gauss0.5 0.5 3 0.175 0.4265 0.5509 1 0.4103 0.9387
C128_l0_clean C128_l0_gauss0.5 1 3 0.1125 0.142 0.3036 0.9107 0.7924 0.9418

Frozen vs fine-tuned (test_top1, mean over seeds)

finetune frozen finetune_minus_frozen
('C128_l0.01_clean', 0.1) 47.9 44.01 3.9
('C128_l0.01_clean', 0.25) 57.55 48.34 9.21
('C128_l0.01_clean', 0.5) 64.12 50.41 13.71
('C128_l0.01_clean', 1.0) 67.9 51.15 16.75
('C128_l0.01_gauss0.5', 0.1) 48.93 42.26 6.67
('C128_l0.01_gauss0.5', 0.25) 57.14 47.79 9.35
('C128_l0.01_gauss0.5', 0.5) 63.78 49.28 14.5
('C128_l0.01_gauss0.5', 1.0) 68.45 50.71 17.74
('C128_l0_clean', 0.1) 49.19 43.9 5.3
('C128_l0_clean', 0.25) 58.44 48.54 9.9
('C128_l0_clean', 0.5) 65.8 50.9 14.9
('C128_l0_clean', 1.0) 68.91 52.55 16.36
('C128_l0_gauss0.5', 0.1) 48.88 42.89 5.99
('C128_l0_gauss0.5', 0.25) 58.58 48.73 9.85
('C128_l0_gauss0.5', 0.5) 65.35 51.08 14.27
('C128_l0_gauss0.5', 1.0) 69.5 52.66 16.84
('C2048_l0.01_clean', 0.1) 49.22 42.94 6.28
('C2048_l0.01_clean', 0.25) 57.58 48.2 9.38
('C2048_l0.01_clean', 0.5) 64.71 50.65 14.06
('C2048_l0.01_clean', 1.0) 67.9 52.08 15.82
('C2048_l0.01_gauss0.5', 0.1) 48.85 43 5.85
('C2048_l0.01_gauss0.5', 0.25) 57.6 48.59 9
('C2048_l0.01_gauss0.5', 0.5) 65.05 51.32 13.73
('C2048_l0.01_gauss0.5', 1.0) 69.51 52.85 16.66
('C2048_l0.01_mask0.5', 0.1) 49.55 45.72 3.83
('C2048_l0.01_mask0.5', 0.25) 59.21 49.67 9.54
('C2048_l0.01_mask0.5', 0.5) 67.4 53.93 13.47
('C2048_l0.01_mask0.5', 1.0) 69.82 55.66 14.15
('C2048_l0_clean', 0.1) 49.77 44.22 5.55
('C2048_l0_clean', 0.25) 55.69 49.39 6.3
('C2048_l0_clean', 0.5) 65.09 52.15 12.95
('C2048_l0_clean', 1.0) 69.55 52.88 16.67
('C2048_l0_gauss0.5', 0.1) 48.93 44.13 4.8
('C2048_l0_gauss0.5', 0.25) 57.85 50.03 7.82
('C2048_l0_gauss0.5', 0.5) 65.55 52.55 13.01
('C2048_l0_gauss0.5', 1.0) 69.77 54.14 15.63

Autoencoder pre-training

variant latent_dim overcomplete_x params_encoder params_decoder train_time_h img_per_s peak_mem_mb holdout_psnr frac_active
oae_l0.001 73728 2.667 12219456 2617155 0.3322 818.7 2550 25.1 0.22
oae_l0.01 73728 2.667 12219456 2617155 0.3115 873.3 2550 25.05 0.1399
oae_l0.1 73728 2.667 12219456 2617155 0.3331 816.6 2550 24.79 0.05403
ae 4608 0.167 11234496 1634115 0.2988 910.4 2507 25.1 0.7798
ae_sparse 4608 0.167 11234496 1634115 0.317 858.2 2507 24.78 0.396
ae_denoise 4608 0.167 11234496 1634115 0.294 925.1 2674 24.74 0.7822
ae_sparse_denoise 4608 0.167 11234496 1634115 0.3021 900.3 2674 24.44 0.3752
oae 73728 2.667 12219456 2617155 0.313 869.1 2550 25.17 0.2507
oae_sparse 73728 2.667 12219456 2617155 0.3115 873.3 2550 25.05 0.1399
oae_denoise 73728 2.667 12219456 2617155 0.2901 937.7 2564 24.79 0.2423
oae_sparse_denoise 73728 2.667 12219456 2617155 0.2947 923.1 2564 24.73 0.1336
oae_sparse_mask 73728 2.667 12219456 2617155 0.2928 928.9 2564 24.01 0.08135

Inference cost (identical downstream architecture for all models)

Parameters: 11,173,962

latency_ms_median latency_ms_p90 throughput_img_s peak_gpu_mem_mb
bs1 3.06 3.21 326 76.4
bs256 45 45.2 5.69e+03 442

Selections (validation only)

{'finetune/ResNet-18 baseline': {'chosen': 'scratch', 'mean_val_top1': {'scratch': np.float64(0.5984999999999999)}}, 'finetune/Standard AE': {'chosen': 'C128_l0_clean', 'mean_val_top1': {'C128_l0_clean': np.float64(0.587)}}, 'finetune/Sparse OAE': {'chosen': 'C2048_l0.01_mask0.5', 'mean_val_top1': {'C2048_l0.01_clean': np.float64(0.5791666666666667), 'C2048_l0.01_gauss0.5': np.float64(0.585), 'C2048_l0.01_mask0.5': np.float64(0.6015)}}, 'frozen/Random frozen ResNet-18 (control)': {'chosen': 'random', 'mean_val_top1': {'random': np.float64(0.418)}}, 'frozen/Standard AE': {'chosen': 'C128_l0_clean', 'mean_val_top1': {'C128_l0_clean': np.float64(0.49733333333333346)}}, 'frozen/Sparse OAE': {'chosen': 'C2048_l0.01_mask0.5', 'mean_val_top1': {'C2048_l0.01_clean': np.float64(0.4971666666666666), 'C2048_l0.01_gauss0.5': np.float64(0.4953333333333334), 'C2048_l0.01_mask0.5': np.float64(0.5216666666666668)}}}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support