metadata tags:
- autoencoder
- sparse-coding
- representation-learning
- self-supervised
- stl10
- resnet18
Overcomplete Sparse Autoencoders for Visual Representation Learning (STL-10)
Controlled study of whether an overcomplete, L1-sparse convolutional autoencoder learns features that transfer
better to STL-10 classification than a conventional undercomplete autoencoder , against a supervised
ResNet-18 trained from scratch , at 10 / 25 / 50 / 100 % of the labelled data.
Code: https://github.com/0x-khalil/OSAE
Shared feature extractor: ResNet-18 adapted to 96×96 (3×3/2 stem, no max-pool); identical downstream classifier for all models.
Standard AE latent: 128×6×6 = 4,608 coefficients (0.17× input). Overcomplete AE latent: 2048×6×6 = 73,728 (2.67× input).
Pre-training on the 100,000 unlabeled images; λ* = 0.01 (selected on validation).
Downstream: full fine-tuning and frozen-encoder protocols, 3 seeds, identical stratified subsets,
validation-only model selection, test split used once for final evaluation.
Files
Path
Content
results/RESULTS.md
headline table, ablations, paired statistics, costs
results/figures/
label-efficiency curves, confusion matrices, per-class accuracy
results/representation/
sparsity, reconstruction, separability, PCA / t-SNE / UMAP, CKA
encoders/<variant>/encoder.pt
pre-trained ResNet-18 trunk (torch.load(...)["encoder"])
downstream/
per-run metrics (and checkpoints if uploaded)
configs/, data_splits/
exact configuration and split indices
Load a pre-trained trunk with the code from the repository:
import torch
from osae.models import Classifier
clf = Classifier()
clf.encoder.load_state_dict(torch.load("encoders/oae_sparse/encoder.pt" )["encoder" ])
Results
All numbers: mean ± std over seeds, in %. Test = official STL-10 test split (8,000 images); every choice below was made on the validation split.
finetune — test_top1
Model
variant
10%
25%
50%
100%
ResNet-18 baseline
scratch
44.11 ± 0.62
54.44 ± 1.86
65.16 ± 0.97
72.77 ± 0.40
Standard AE
C128_l0_clean
49.19 ± 0.28
58.44 ± 0.29
65.80 ± 0.12
68.91 ± 0.64
Sparse OAE
C2048_l0.01_mask0.5
49.55 ± 0.84
59.21 ± 0.54
67.40 ± 0.14
69.82 ± 4.37
finetune — test_macro_f1
Model
variant
10%
25%
50%
100%
ResNet-18 baseline
scratch
44.00 ± 0.65
54.40 ± 1.93
65.21 ± 0.97
72.87 ± 0.39
Standard AE
C128_l0_clean
49.02 ± 0.28
58.46 ± 0.28
65.84 ± 0.16
68.81 ± 0.77
Sparse OAE
C2048_l0.01_mask0.5
49.36 ± 0.82
59.16 ± 0.46
67.45 ± 0.15
69.76 ± 4.46
finetune — val_top1
Model
variant
10%
25%
50%
100%
ResNet-18 baseline
scratch
47.47 ± 2.31
57.20 ± 1.22
64.40 ± 1.59
70.33 ± 1.10
Standard AE
C128_l0_clean
51.13 ± 0.42
57.07 ± 0.95
62.27 ± 1.50
64.33 ± 1.30
Sparse OAE
C2048_l0.01_mask0.5
50.40 ± 1.83
59.33 ± 0.31
64.07 ± 0.70
66.80 ± 0.53
finetune — all variants (test_top1)
variant
latent
lambda
noise
10%
25%
50%
100%
val (all f)
ae_sparse
128
0.01
none
47.90 ± 0.87
57.55 ± 0.64
64.12 ± 0.63
67.90 ± 0.79
59.02
ae_sparse_denoise
128
0.01
gaussian
48.93 ± 0.15
57.14 ± 0.50
63.78 ± 0.94
68.45 ± 0.51
59
ae
128
0.0
none
49.19 ± 0.28
58.44 ± 0.29
65.80 ± 0.12
68.91 ± 0.64
58.7
ae_denoise
128
0.0
gaussian
48.88 ± 0.48
58.58 ± 0.24
65.35 ± 0.73
69.50 ± 0.35
59.17
oae_l0.01/oae_sparse
2048
0.01
none
49.22 ± 0.51
57.58 ± 0.64
64.71 ± 0.39
67.90 ± 0.99
57.92
oae_sparse_denoise
2048
0.01
gaussian
48.85 ± 0.66
57.60 ± 0.53
65.05 ± 0.52
69.51 ± 0.14
58.5
oae_sparse_mask
2048
0.01
mask
49.55 ± 0.84
59.21 ± 0.54
67.40 ± 0.14
69.82 ± 4.37
60.15
oae
2048
0.0
none
49.77 ± 0.46
55.69 ± 1.58
65.09 ± 0.33
69.55 ± 0.48
58.33
oae_denoise
2048
0.0
gaussian
48.93 ± 0.40
57.85 ± 0.46
65.55 ± 0.16
69.77 ± 0.21
59.35
scratch
–
–
–
44.11 ± 0.62
54.44 ± 1.86
65.16 ± 0.97
72.77 ± 0.40
59.85
finetune — downstream training cost (mean over runs)
canonical
train_time_s
peak_train_mem_mb
params_total
params_trainable
C128_l0.01_clean
147.5
919.2
1.1174e+07
1.1174e+07
C128_l0.01_gauss0.5
145.1
919.2
1.1174e+07
1.1174e+07
C128_l0_clean
153
919.2
1.1174e+07
1.1174e+07
C128_l0_gauss0.5
149.9
919.2
1.1174e+07
1.1174e+07
C2048_l0.01_clean
148.4
919.2
1.1174e+07
1.1174e+07
C2048_l0.01_gauss0.5
142
919.2
1.1174e+07
1.1174e+07
C2048_l0.01_mask0.5
143.9
919.2
1.1174e+07
1.1174e+07
C2048_l0_clean
148.7
919.2
1.1174e+07
1.1174e+07
C2048_l0_gauss0.5
147.4
919.2
1.1174e+07
1.1174e+07
scratch
151
919.2
1.1174e+07
1.1174e+07
finetune — paired comparisons (test top-1, B − A in points)
A
B
fraction
n_seeds
diff_B_minus_A
diff_std
p_paired_t
p_paired_t_holm
cohens_dz
p_mcnemar_max_over_seeds
scratch
C128_l0_clean
0.1
3
5.083
0.719
0.006602
0.02641
7.07
2.226e-14
scratch
C128_l0_clean
0.25
3
4.004
2.073
0.07893
0.1579
1.931
0.002828
scratch
C128_l0_clean
0.5
3
0.6417
1.079
0.4112
0.4112
0.5947
0.3509
scratch
C128_l0_clean
1
3
-3.858
0.5837
0.007542
0.02641
-6.61
7.597e-11
scratch
C2048_l0.01_mask0.5
0.1
3
5.438
0.9135
0.009278
0.03711
5.952
6.667e-17
scratch
C2048_l0.01_mask0.5
0.25
3
4.775
1.338
0.0252
0.07559
3.568
2.118e-09
scratch
C2048_l0.01_mask0.5
0.5
3
2.237
1.038
0.06488
0.1298
2.155
0.01869
scratch
C2048_l0.01_mask0.5
1
3
-2.95
4.179
0.346
0.346
-0.7059
0.5993
C128_l0_clean
C2048_l0.01_mask0.5
0.1
3
0.3542
1.122
0.6394
1
0.3157
0.3909
C128_l0_clean
C2048_l0.01_mask0.5
0.25
3
0.7708
0.8079
0.2402
0.7207
0.9541
0.7619
C128_l0_clean
C2048_l0.01_mask0.5
0.5
3
1.596
0.1938
0.004879
0.01952
8.235
0.004674
C128_l0_clean
C2048_l0.01_mask0.5
1
3
0.9083
3.722
0.7136
1
0.244
4.28e-09
C128_l0_clean
C2048_l0_clean
0.1
3
0.5792
0.6411
0.2581
0.3788
0.9034
0.8093
C128_l0_clean
C2048_l0_clean
0.25
3
-2.75
1.875
0.1263
0.3788
-1.467
0.05449
C128_l0_clean
C2048_l0_clean
0.5
3
-0.7125
0.219
0.03008
0.1203
-3.253
0.2517
C128_l0_clean
C2048_l0_clean
1
3
0.6375
0.553
0.1839
0.3788
1.153
1
C2048_l0_clean
C2048_l0.01_clean
0.1
3
-0.55
0.2382
0.05719
0.2288
-2.309
0.3679
C2048_l0_clean
C2048_l0.01_clean
0.25
3
1.887
1.003
0.08257
0.2477
1.883
0.01547
C2048_l0_clean
C2048_l0.01_clean
0.5
3
-0.3792
0.7142
0.4549
0.4549
-0.5309
0.3409
C2048_l0_clean
C2048_l0.01_clean
1
3
-1.646
1.06
0.1149
0.2477
-1.553
0.1841
C128_l0_clean
C128_l0.01_clean
0.1
3
-1.288
0.7253
0.0915
0.2745
-1.775
0.1862
C128_l0_clean
C128_l0.01_clean
0.25
3
-0.8917
0.7381
0.1715
0.343
-1.208
0.7835
C128_l0_clean
C128_l0.01_clean
0.5
3
-1.683
0.5693
0.03608
0.1443
-2.957
0.01671
C128_l0_clean
C128_l0.01_clean
1
3
-1.008
1.264
0.3012
0.343
-0.7977
0.4219
C2048_l0.01_clean
C2048_l0.01_gauss0.5
0.1
3
-0.3708
0.4537
0.2925
0.8775
-0.8174
0.824
C2048_l0.01_clean
C2048_l0.01_gauss0.5
0.25
3
0.01667
1.096
0.9814
0.9814
0.01521
0.2753
C2048_l0.01_clean
C2048_l0.01_gauss0.5
0.5
3
0.3417
0.6825
0.4773
0.9546
0.5006
0.6171
C2048_l0.01_clean
C2048_l0.01_gauss0.5
1
3
1.613
0.8928
0.08878
0.3551
1.806
0.02627
C128_l0_clean
C128_l0_gauss0.5
0.1
3
-0.3167
0.3224
0.231
0.9241
-0.9821
0.9317
C128_l0_clean
C128_l0_gauss0.5
0.25
3
0.1417
0.4161
0.6152
0.9852
0.3404
0.4568
C128_l0_clean
C128_l0_gauss0.5
0.5
3
-0.4583
0.6193
0.3284
0.9852
-0.7401
0.6854
C128_l0_clean
C128_l0_gauss0.5
1
3
0.5917
0.9863
0.4079
0.9852
0.5999
0.5875
frozen — test_top1
Model
variant
10%
25%
50%
100%
Random frozen ResNet-18 (control)
random
35.58 ± 0.28
39.48 ± 0.34
41.61 ± 0.72
42.82 ± 1.13
Standard AE
C128_l0_clean
43.90 ± 0.29
48.54 ± 1.39
50.90 ± 0.56
52.55 ± 0.21
Sparse OAE
C2048_l0.01_mask0.5
45.72 ± 0.18
49.67 ± 1.44
53.93 ± 0.09
55.66 ± 0.30
frozen — test_macro_f1
Model
variant
10%
25%
50%
100%
Random frozen ResNet-18 (control)
random
35.26 ± 0.24
39.22 ± 0.38
41.17 ± 0.66
42.36 ± 1.14
Standard AE
C128_l0_clean
43.83 ± 0.30
48.31 ± 1.34
50.68 ± 0.62
52.26 ± 0.26
Sparse OAE
C2048_l0.01_mask0.5
45.50 ± 0.19
49.52 ± 1.48
53.86 ± 0.08
55.54 ± 0.33
frozen — val_top1
Model
variant
10%
25%
50%
100%
Random frozen ResNet-18 (control)
random
37.20 ± 1.56
41.93 ± 1.01
42.80 ± 3.33
45.27 ± 2.34
Standard AE
C128_l0_clean
45.60 ± 0.20
50.27 ± 0.81
50.73 ± 1.14
52.33 ± 1.21
Sparse OAE
C2048_l0.01_mask0.5
46.33 ± 0.50
51.20 ± 0.20
54.53 ± 0.83
56.60 ± 0.53
frozen — all variants (test_top1)
variant
latent
lambda
noise
10%
25%
50%
100%
val (all f)
ae_sparse
128
0.01
none
44.01 ± 0.35
48.34 ± 0.25
50.41 ± 0.13
51.15 ± 0.36
49
ae_sparse_denoise
128
0.01
gaussian
42.26 ± 0.89
47.79 ± 0.43
49.28 ± 0.30
50.71 ± 0.29
47.5
ae
128
0.0
none
43.90 ± 0.29
48.54 ± 1.39
50.90 ± 0.56
52.55 ± 0.21
49.73
ae_denoise
128
0.0
gaussian
42.89 ± 0.56
48.73 ± 0.47
51.08 ± 0.47
52.66 ± 0.31
48.43
oae_l0.001
2048
0.001
none
44.61
–
–
53.02
48.3
oae_l0.01/oae_sparse
2048
0.01
none
42.94 ± 0.62
48.20 ± 0.14
50.65 ± 0.57
52.07 ± 0.42
49.72
oae_sparse_denoise
2048
0.01
gaussian
43.00 ± 0.61
48.59 ± 1.46
51.33 ± 0.41
52.85 ± 0.26
49.53
oae_sparse_mask
2048
0.01
mask
45.72 ± 0.18
49.67 ± 1.44
53.93 ± 0.09
55.66 ± 0.30
52.17
oae_l0.1
2048
0.1
none
42.27
–
–
50.56
46.5
oae
2048
0.0
none
44.22 ± 0.91
49.39 ± 0.86
52.15 ± 0.20
52.88 ± 0.85
51.03
oae_denoise
2048
0.0
gaussian
44.13 ± 0.45
50.03 ± 0.24
52.55 ± 0.27
54.14 ± 0.10
50.67
random
–
–
–
35.58 ± 0.28
39.48 ± 0.34
41.61 ± 0.72
42.82 ± 1.13
41.8
frozen — downstream training cost (mean over runs)
canonical
train_time_s
peak_train_mem_mb
params_total
params_trainable
C128_l0.01_clean
59.8
832.6
1.1174e+07
5130
C128_l0.01_gauss0.5
57.3
832.6
1.1174e+07
5130
C128_l0_clean
56.6
832.6
1.1174e+07
5130
C128_l0_gauss0.5
59.4
832.6
1.1174e+07
5130
C2048_l0.001_clean
64.2
832.6
1.1174e+07
5130
C2048_l0.01_clean
56.5
832.6
1.1174e+07
5130
C2048_l0.01_gauss0.5
56.6
832.6
1.1174e+07
5130
C2048_l0.01_mask0.5
57.9
832.6
1.1174e+07
5130
C2048_l0.1_clean
66.5
832.6
1.1174e+07
5130
C2048_l0_clean
61
832.6
1.1174e+07
5130
C2048_l0_gauss0.5
56.5
832.6
1.1174e+07
5130
random
232.8
2418.4
1.1174e+07
5130
frozen — paired comparisons (test top-1, B − A in points)
A
B
fraction
n_seeds
diff_B_minus_A
diff_std
p_paired_t
p_paired_t_holm
cohens_dz
p_mcnemar_max_over_seeds
C128_l0_clean
C2048_l0.01_mask0.5
0.1
3
1.825
0.1146
0.001311
0.005244
15.93
0.0018
C128_l0_clean
C2048_l0.01_mask0.5
0.25
3
1.133
2.788
0.5543
0.5543
0.4066
0.0002473
C128_l0_clean
C2048_l0.01_mask0.5
0.5
3
3.029
0.5236
0.009812
0.02457
5.786
6.927e-06
C128_l0_clean
C2048_l0.01_mask0.5
1
3
3.117
0.4915
0.00819
0.02457
6.341
9.169e-07
C128_l0_clean
C2048_l0_clean
0.1
3
0.3208
0.657
0.4867
1
0.4884
0.8107
C128_l0_clean
C2048_l0_clean
0.25
3
0.85
2.234
0.5776
1
0.3805
0.0009237
C128_l0_clean
C2048_l0_clean
0.5
3
1.242
0.5646
0.06253
0.2501
2.199
0.2492
C128_l0_clean
C2048_l0_clean
1
3
0.3333
0.7821
0.5372
1
0.4262
0.2995
C2048_l0_clean
C2048_l0.01_clean
0.1
3
-1.279
0.8055
0.1107
0.332
-1.588
0.4022
C2048_l0_clean
C2048_l0.01_clean
0.25
3
-1.192
0.9306
0.1568
0.332
-1.281
0.8221
C2048_l0_clean
C2048_l0.01_clean
0.5
3
-1.496
0.6377
0.05557
0.2223
-2.346
0.09288
C2048_l0_clean
C2048_l0.01_clean
1
3
-0.8042
1.172
0.3567
0.3567
-0.6861
0.3488
C128_l0_clean
C128_l0.01_clean
0.1
3
0.1125
0.5717
0.7657
1
0.1968
0.5233
C128_l0_clean
C128_l0.01_clean
0.25
3
-0.2
1.61
0.8496
1
-0.1242
0.2949
C128_l0_clean
C128_l0.01_clean
0.5
3
-0.4958
0.6831
0.3356
1
-0.7259
1
C128_l0_clean
C128_l0.01_clean
1
3
-1.396
0.2306
0.008975
0.0359
-6.053
0.01661
C2048_l0.01_clean
C2048_l0.01_gauss0.5
0.1
3
0.0625
0.598
0.873
1
0.1045
0.5002
C2048_l0.01_clean
C2048_l0.01_gauss0.5
0.25
3
0.3958
1.325
0.6564
1
0.2988
0.05433
C2048_l0.01_clean
C2048_l0.01_gauss0.5
0.5
3
0.675
0.3356
0.07345
0.2938
2.011
0.5933
C2048_l0.01_clean
C2048_l0.01_gauss0.5
1
3
0.7792
0.4661
0.1014
0.3043
1.672
0.5116
C128_l0_clean
C128_l0_gauss0.5
0.1
3
-1.008
0.593
0.09853
0.3941
-1.7
0.5056
C128_l0_clean
C128_l0_gauss0.5
0.25
3
0.1958
0.9284
0.7499
1
0.2109
0.203
C128_l0_clean
C128_l0_gauss0.5
0.5
3
0.175
0.4265
0.5509
1
0.4103
0.9387
C128_l0_clean
C128_l0_gauss0.5
1
3
0.1125
0.142
0.3036
0.9107
0.7924
0.9418
Frozen vs fine-tuned (test_top1, mean over seeds)
finetune
frozen
finetune_minus_frozen
('C128_l0.01_clean', 0.1)
47.9
44.01
3.9
('C128_l0.01_clean', 0.25)
57.55
48.34
9.21
('C128_l0.01_clean', 0.5)
64.12
50.41
13.71
('C128_l0.01_clean', 1.0)
67.9
51.15
16.75
('C128_l0.01_gauss0.5', 0.1)
48.93
42.26
6.67
('C128_l0.01_gauss0.5', 0.25)
57.14
47.79
9.35
('C128_l0.01_gauss0.5', 0.5)
63.78
49.28
14.5
('C128_l0.01_gauss0.5', 1.0)
68.45
50.71
17.74
('C128_l0_clean', 0.1)
49.19
43.9
5.3
('C128_l0_clean', 0.25)
58.44
48.54
9.9
('C128_l0_clean', 0.5)
65.8
50.9
14.9
('C128_l0_clean', 1.0)
68.91
52.55
16.36
('C128_l0_gauss0.5', 0.1)
48.88
42.89
5.99
('C128_l0_gauss0.5', 0.25)
58.58
48.73
9.85
('C128_l0_gauss0.5', 0.5)
65.35
51.08
14.27
('C128_l0_gauss0.5', 1.0)
69.5
52.66
16.84
('C2048_l0.01_clean', 0.1)
49.22
42.94
6.28
('C2048_l0.01_clean', 0.25)
57.58
48.2
9.38
('C2048_l0.01_clean', 0.5)
64.71
50.65
14.06
('C2048_l0.01_clean', 1.0)
67.9
52.08
15.82
('C2048_l0.01_gauss0.5', 0.1)
48.85
43
5.85
('C2048_l0.01_gauss0.5', 0.25)
57.6
48.59
9
('C2048_l0.01_gauss0.5', 0.5)
65.05
51.32
13.73
('C2048_l0.01_gauss0.5', 1.0)
69.51
52.85
16.66
('C2048_l0.01_mask0.5', 0.1)
49.55
45.72
3.83
('C2048_l0.01_mask0.5', 0.25)
59.21
49.67
9.54
('C2048_l0.01_mask0.5', 0.5)
67.4
53.93
13.47
('C2048_l0.01_mask0.5', 1.0)
69.82
55.66
14.15
('C2048_l0_clean', 0.1)
49.77
44.22
5.55
('C2048_l0_clean', 0.25)
55.69
49.39
6.3
('C2048_l0_clean', 0.5)
65.09
52.15
12.95
('C2048_l0_clean', 1.0)
69.55
52.88
16.67
('C2048_l0_gauss0.5', 0.1)
48.93
44.13
4.8
('C2048_l0_gauss0.5', 0.25)
57.85
50.03
7.82
('C2048_l0_gauss0.5', 0.5)
65.55
52.55
13.01
('C2048_l0_gauss0.5', 1.0)
69.77
54.14
15.63
Autoencoder pre-training
variant
latent_dim
overcomplete_x
params_encoder
params_decoder
train_time_h
img_per_s
peak_mem_mb
holdout_psnr
frac_active
oae_l0.001
73728
2.667
12219456
2617155
0.3322
818.7
2550
25.1
0.22
oae_l0.01
73728
2.667
12219456
2617155
0.3115
873.3
2550
25.05
0.1399
oae_l0.1
73728
2.667
12219456
2617155
0.3331
816.6
2550
24.79
0.05403
ae
4608
0.167
11234496
1634115
0.2988
910.4
2507
25.1
0.7798
ae_sparse
4608
0.167
11234496
1634115
0.317
858.2
2507
24.78
0.396
ae_denoise
4608
0.167
11234496
1634115
0.294
925.1
2674
24.74
0.7822
ae_sparse_denoise
4608
0.167
11234496
1634115
0.3021
900.3
2674
24.44
0.3752
oae
73728
2.667
12219456
2617155
0.313
869.1
2550
25.17
0.2507
oae_sparse
73728
2.667
12219456
2617155
0.3115
873.3
2550
25.05
0.1399
oae_denoise
73728
2.667
12219456
2617155
0.2901
937.7
2564
24.79
0.2423
oae_sparse_denoise
73728
2.667
12219456
2617155
0.2947
923.1
2564
24.73
0.1336
oae_sparse_mask
73728
2.667
12219456
2617155
0.2928
928.9
2564
24.01
0.08135
Inference cost (identical downstream architecture for all models)
Parameters: 11,173,962
latency_ms_median
latency_ms_p90
throughput_img_s
peak_gpu_mem_mb
bs1
3.06
3.21
326
76.4
bs256
45
45.2
5.69e+03
442
Selections (validation only)
{'finetune/ResNet-18 baseline': {'chosen': 'scratch', 'mean_val_top1': {'scratch': np.float64(0.5984999999999999)}}, 'finetune/Standard AE': {'chosen': 'C128_l0_clean', 'mean_val_top1': {'C128_l0_clean': np.float64(0.587)}}, 'finetune/Sparse OAE': {'chosen': 'C2048_l0.01_mask0.5', 'mean_val_top1': {'C2048_l0.01_clean': np.float64(0.5791666666666667), 'C2048_l0.01_gauss0.5': np.float64(0.585), 'C2048_l0.01_mask0.5': np.float64(0.6015)}}, 'frozen/Random frozen ResNet-18 (control)': {'chosen': 'random', 'mean_val_top1': {'random': np.float64(0.418)}}, 'frozen/Standard AE': {'chosen': 'C128_l0_clean', 'mean_val_top1': {'C128_l0_clean': np.float64(0.49733333333333346)}}, 'frozen/Sparse OAE': {'chosen': 'C2048_l0.01_mask0.5', 'mean_val_top1': {'C2048_l0.01_clean': np.float64(0.4971666666666666), 'C2048_l0.01_gauss0.5': np.float64(0.4953333333333334), 'C2048_l0.01_mask0.5': np.float64(0.5216666666666668)}}}