OSAE / README.md
0x-Khalil's picture
Upload OSAE study results
7ce4d7d verified
|
Raw History Blame Contribute Delete
30.2 kB
---
tags:
- autoencoder
- sparse-coding
- representation-learning
- self-supervised
- stl10
- resnet18
---
# Overcomplete Sparse Autoencoders for Visual Representation Learning (STL-10)
Controlled study of whether an **overcomplete, L1-sparse convolutional autoencoder** learns features that transfer
better to STL-10 classification than a **conventional undercomplete autoencoder**, against a **supervised
ResNet-18 trained from scratch**, at 10 / 25 / 50 / 100 % of the labelled data.
Code: https://github.com/0x-khalil/OSAE
* Shared feature extractor: ResNet-18 adapted to 96×96 (3×3/2 stem, no max-pool); identical downstream classifier for all models.
* Standard AE latent: 128×6×6 = 4,608 coefficients (0.17× input). Overcomplete AE latent: 2048×6×6 = 73,728 (2.67× input).
* Pre-training on the 100,000 unlabeled images; λ* = 0.01 (selected on validation).
* Downstream: full fine-tuning and frozen-encoder protocols, 3 seeds, identical stratified subsets,
validation-only model selection, test split used once for final evaluation.
## Files
| Path | Content |
|---|---|
| `results/RESULTS.md` | headline table, ablations, paired statistics, costs |
| `results/figures/` | label-efficiency curves, confusion matrices, per-class accuracy |
| `results/representation/` | sparsity, reconstruction, separability, PCA / t-SNE / UMAP, CKA |
| `encoders/<variant>/encoder.pt` | pre-trained ResNet-18 trunk (`torch.load(...)["encoder"]`) |
| `downstream/` | per-run metrics (and checkpoints if uploaded) |
| `configs/`, `data_splits/` | exact configuration and split indices |
Load a pre-trained trunk with the code from the repository:
```python
import torch
from osae.models import Classifier
clf = Classifier()
clf.encoder.load_state_dict(torch.load("encoders/oae_sparse/encoder.pt")["encoder"])
```
---
# Results
All numbers: mean ± std over seeds, in %. Test = official STL-10 test split (8,000 images); every choice below was made on the validation split.
## finetune — test_top1
| Model | variant | 10% | 25% | 50% | 100% |
|:-------------------|:--------------------|:-------------|:-------------|:-------------|:-------------|
| ResNet-18 baseline | scratch | 44.11 ± 0.62 | 54.44 ± 1.86 | 65.16 ± 0.97 | 72.77 ± 0.40 |
| Standard AE | C128_l0_clean | 49.19 ± 0.28 | 58.44 ± 0.29 | 65.80 ± 0.12 | 68.91 ± 0.64 |
| Sparse OAE | C2048_l0.01_mask0.5 | 49.55 ± 0.84 | 59.21 ± 0.54 | 67.40 ± 0.14 | 69.82 ± 4.37 |
## finetune — test_macro_f1
| Model | variant | 10% | 25% | 50% | 100% |
|:-------------------|:--------------------|:-------------|:-------------|:-------------|:-------------|
| ResNet-18 baseline | scratch | 44.00 ± 0.65 | 54.40 ± 1.93 | 65.21 ± 0.97 | 72.87 ± 0.39 |
| Standard AE | C128_l0_clean | 49.02 ± 0.28 | 58.46 ± 0.28 | 65.84 ± 0.16 | 68.81 ± 0.77 |
| Sparse OAE | C2048_l0.01_mask0.5 | 49.36 ± 0.82 | 59.16 ± 0.46 | 67.45 ± 0.15 | 69.76 ± 4.46 |
## finetune — val_top1
| Model | variant | 10% | 25% | 50% | 100% |
|:-------------------|:--------------------|:-------------|:-------------|:-------------|:-------------|
| ResNet-18 baseline | scratch | 47.47 ± 2.31 | 57.20 ± 1.22 | 64.40 ± 1.59 | 70.33 ± 1.10 |
| Standard AE | C128_l0_clean | 51.13 ± 0.42 | 57.07 ± 0.95 | 62.27 ± 1.50 | 64.33 ± 1.30 |
| Sparse OAE | C2048_l0.01_mask0.5 | 50.40 ± 1.83 | 59.33 ± 0.31 | 64.07 ± 0.70 | 66.80 ± 0.53 |
## finetune — all variants (test_top1)
| variant | latent | lambda | noise | 10% | 25% | 50% | 100% | val (all f) |
|:---------------------|:---------|:---------|:---------|:-------------|:-------------|:-------------|:-------------|--------------:|
| ae_sparse | 128 | 0.01 | none | 47.90 ± 0.87 | 57.55 ± 0.64 | 64.12 ± 0.63 | 67.90 ± 0.79 | 59.02 |
| ae_sparse_denoise | 128 | 0.01 | gaussian | 48.93 ± 0.15 | 57.14 ± 0.50 | 63.78 ± 0.94 | 68.45 ± 0.51 | 59 |
| ae | 128 | 0.0 | none | 49.19 ± 0.28 | 58.44 ± 0.29 | 65.80 ± 0.12 | 68.91 ± 0.64 | 58.7 |
| ae_denoise | 128 | 0.0 | gaussian | 48.88 ± 0.48 | 58.58 ± 0.24 | 65.35 ± 0.73 | 69.50 ± 0.35 | 59.17 |
| oae_l0.01/oae_sparse | 2048 | 0.01 | none | 49.22 ± 0.51 | 57.58 ± 0.64 | 64.71 ± 0.39 | 67.90 ± 0.99 | 57.92 |
| oae_sparse_denoise | 2048 | 0.01 | gaussian | 48.85 ± 0.66 | 57.60 ± 0.53 | 65.05 ± 0.52 | 69.51 ± 0.14 | 58.5 |
| oae_sparse_mask | 2048 | 0.01 | mask | 49.55 ± 0.84 | 59.21 ± 0.54 | 67.40 ± 0.14 | 69.82 ± 4.37 | 60.15 |
| oae | 2048 | 0.0 | none | 49.77 ± 0.46 | 55.69 ± 1.58 | 65.09 ± 0.33 | 69.55 ± 0.48 | 58.33 |
| oae_denoise | 2048 | 0.0 | gaussian | 48.93 ± 0.40 | 57.85 ± 0.46 | 65.55 ± 0.16 | 69.77 ± 0.21 | 59.35 |
| scratch | – | – | – | 44.11 ± 0.62 | 54.44 ± 1.86 | 65.16 ± 0.97 | 72.77 ± 0.40 | 59.85 |
## finetune — downstream training cost (mean over runs)
| canonical | train_time_s | peak_train_mem_mb | params_total | params_trainable |
|:---------------------|---------------:|--------------------:|---------------:|-------------------:|
| C128_l0.01_clean | 147.5 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| C128_l0.01_gauss0.5 | 145.1 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| C128_l0_clean | 153 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| C128_l0_gauss0.5 | 149.9 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| C2048_l0.01_clean | 148.4 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| C2048_l0.01_gauss0.5 | 142 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| C2048_l0.01_mask0.5 | 143.9 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| C2048_l0_clean | 148.7 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| C2048_l0_gauss0.5 | 147.4 | 919.2 | 1.1174e+07 | 1.1174e+07 |
| scratch | 151 | 919.2 | 1.1174e+07 | 1.1174e+07 |
## finetune — paired comparisons (test top-1, B − A in points)
| A | B | fraction | n_seeds | diff_B_minus_A | diff_std | p_paired_t | p_paired_t_holm | cohens_dz | p_mcnemar_max_over_seeds |
|:------------------|:---------------------|-----------:|----------:|-----------------:|-----------:|-------------:|------------------:|------------:|---------------------------:|
| scratch | C128_l0_clean | 0.1 | 3 | 5.083 | 0.719 | 0.006602 | 0.02641 | 7.07 | 2.226e-14 |
| scratch | C128_l0_clean | 0.25 | 3 | 4.004 | 2.073 | 0.07893 | 0.1579 | 1.931 | 0.002828 |
| scratch | C128_l0_clean | 0.5 | 3 | 0.6417 | 1.079 | 0.4112 | 0.4112 | 0.5947 | 0.3509 |
| scratch | C128_l0_clean | 1 | 3 | -3.858 | 0.5837 | 0.007542 | 0.02641 | -6.61 | 7.597e-11 |
| scratch | C2048_l0.01_mask0.5 | 0.1 | 3 | 5.438 | 0.9135 | 0.009278 | 0.03711 | 5.952 | 6.667e-17 |
| scratch | C2048_l0.01_mask0.5 | 0.25 | 3 | 4.775 | 1.338 | 0.0252 | 0.07559 | 3.568 | 2.118e-09 |
| scratch | C2048_l0.01_mask0.5 | 0.5 | 3 | 2.237 | 1.038 | 0.06488 | 0.1298 | 2.155 | 0.01869 |
| scratch | C2048_l0.01_mask0.5 | 1 | 3 | -2.95 | 4.179 | 0.346 | 0.346 | -0.7059 | 0.5993 |
| C128_l0_clean | C2048_l0.01_mask0.5 | 0.1 | 3 | 0.3542 | 1.122 | 0.6394 | 1 | 0.3157 | 0.3909 |
| C128_l0_clean | C2048_l0.01_mask0.5 | 0.25 | 3 | 0.7708 | 0.8079 | 0.2402 | 0.7207 | 0.9541 | 0.7619 |
| C128_l0_clean | C2048_l0.01_mask0.5 | 0.5 | 3 | 1.596 | 0.1938 | 0.004879 | 0.01952 | 8.235 | 0.004674 |
| C128_l0_clean | C2048_l0.01_mask0.5 | 1 | 3 | 0.9083 | 3.722 | 0.7136 | 1 | 0.244 | 4.28e-09 |
| C128_l0_clean | C2048_l0_clean | 0.1 | 3 | 0.5792 | 0.6411 | 0.2581 | 0.3788 | 0.9034 | 0.8093 |
| C128_l0_clean | C2048_l0_clean | 0.25 | 3 | -2.75 | 1.875 | 0.1263 | 0.3788 | -1.467 | 0.05449 |
| C128_l0_clean | C2048_l0_clean | 0.5 | 3 | -0.7125 | 0.219 | 0.03008 | 0.1203 | -3.253 | 0.2517 |
| C128_l0_clean | C2048_l0_clean | 1 | 3 | 0.6375 | 0.553 | 0.1839 | 0.3788 | 1.153 | 1 |
| C2048_l0_clean | C2048_l0.01_clean | 0.1 | 3 | -0.55 | 0.2382 | 0.05719 | 0.2288 | -2.309 | 0.3679 |
| C2048_l0_clean | C2048_l0.01_clean | 0.25 | 3 | 1.887 | 1.003 | 0.08257 | 0.2477 | 1.883 | 0.01547 |
| C2048_l0_clean | C2048_l0.01_clean | 0.5 | 3 | -0.3792 | 0.7142 | 0.4549 | 0.4549 | -0.5309 | 0.3409 |
| C2048_l0_clean | C2048_l0.01_clean | 1 | 3 | -1.646 | 1.06 | 0.1149 | 0.2477 | -1.553 | 0.1841 |
| C128_l0_clean | C128_l0.01_clean | 0.1 | 3 | -1.288 | 0.7253 | 0.0915 | 0.2745 | -1.775 | 0.1862 |
| C128_l0_clean | C128_l0.01_clean | 0.25 | 3 | -0.8917 | 0.7381 | 0.1715 | 0.343 | -1.208 | 0.7835 |
| C128_l0_clean | C128_l0.01_clean | 0.5 | 3 | -1.683 | 0.5693 | 0.03608 | 0.1443 | -2.957 | 0.01671 |
| C128_l0_clean | C128_l0.01_clean | 1 | 3 | -1.008 | 1.264 | 0.3012 | 0.343 | -0.7977 | 0.4219 |
| C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.1 | 3 | -0.3708 | 0.4537 | 0.2925 | 0.8775 | -0.8174 | 0.824 |
| C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.25 | 3 | 0.01667 | 1.096 | 0.9814 | 0.9814 | 0.01521 | 0.2753 |
| C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.5 | 3 | 0.3417 | 0.6825 | 0.4773 | 0.9546 | 0.5006 | 0.6171 |
| C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 1 | 3 | 1.613 | 0.8928 | 0.08878 | 0.3551 | 1.806 | 0.02627 |
| C128_l0_clean | C128_l0_gauss0.5 | 0.1 | 3 | -0.3167 | 0.3224 | 0.231 | 0.9241 | -0.9821 | 0.9317 |
| C128_l0_clean | C128_l0_gauss0.5 | 0.25 | 3 | 0.1417 | 0.4161 | 0.6152 | 0.9852 | 0.3404 | 0.4568 |
| C128_l0_clean | C128_l0_gauss0.5 | 0.5 | 3 | -0.4583 | 0.6193 | 0.3284 | 0.9852 | -0.7401 | 0.6854 |
| C128_l0_clean | C128_l0_gauss0.5 | 1 | 3 | 0.5917 | 0.9863 | 0.4079 | 0.9852 | 0.5999 | 0.5875 |
## frozen — test_top1
| Model | variant | 10% | 25% | 50% | 100% |
|:----------------------------------|:--------------------|:-------------|:-------------|:-------------|:-------------|
| Random frozen ResNet-18 (control) | random | 35.58 ± 0.28 | 39.48 ± 0.34 | 41.61 ± 0.72 | 42.82 ± 1.13 |
| Standard AE | C128_l0_clean | 43.90 ± 0.29 | 48.54 ± 1.39 | 50.90 ± 0.56 | 52.55 ± 0.21 |
| Sparse OAE | C2048_l0.01_mask0.5 | 45.72 ± 0.18 | 49.67 ± 1.44 | 53.93 ± 0.09 | 55.66 ± 0.30 |
## frozen — test_macro_f1
| Model | variant | 10% | 25% | 50% | 100% |
|:----------------------------------|:--------------------|:-------------|:-------------|:-------------|:-------------|
| Random frozen ResNet-18 (control) | random | 35.26 ± 0.24 | 39.22 ± 0.38 | 41.17 ± 0.66 | 42.36 ± 1.14 |
| Standard AE | C128_l0_clean | 43.83 ± 0.30 | 48.31 ± 1.34 | 50.68 ± 0.62 | 52.26 ± 0.26 |
| Sparse OAE | C2048_l0.01_mask0.5 | 45.50 ± 0.19 | 49.52 ± 1.48 | 53.86 ± 0.08 | 55.54 ± 0.33 |
## frozen — val_top1
| Model | variant | 10% | 25% | 50% | 100% |
|:----------------------------------|:--------------------|:-------------|:-------------|:-------------|:-------------|
| Random frozen ResNet-18 (control) | random | 37.20 ± 1.56 | 41.93 ± 1.01 | 42.80 ± 3.33 | 45.27 ± 2.34 |
| Standard AE | C128_l0_clean | 45.60 ± 0.20 | 50.27 ± 0.81 | 50.73 ± 1.14 | 52.33 ± 1.21 |
| Sparse OAE | C2048_l0.01_mask0.5 | 46.33 ± 0.50 | 51.20 ± 0.20 | 54.53 ± 0.83 | 56.60 ± 0.53 |
## frozen — all variants (test_top1)
| variant | latent | lambda | noise | 10% | 25% | 50% | 100% | val (all f) |
|:---------------------|:---------|:---------|:---------|:-------------|:-------------|:-------------|:-------------|--------------:|
| ae_sparse | 128 | 0.01 | none | 44.01 ± 0.35 | 48.34 ± 0.25 | 50.41 ± 0.13 | 51.15 ± 0.36 | 49 |
| ae_sparse_denoise | 128 | 0.01 | gaussian | 42.26 ± 0.89 | 47.79 ± 0.43 | 49.28 ± 0.30 | 50.71 ± 0.29 | 47.5 |
| ae | 128 | 0.0 | none | 43.90 ± 0.29 | 48.54 ± 1.39 | 50.90 ± 0.56 | 52.55 ± 0.21 | 49.73 |
| ae_denoise | 128 | 0.0 | gaussian | 42.89 ± 0.56 | 48.73 ± 0.47 | 51.08 ± 0.47 | 52.66 ± 0.31 | 48.43 |
| oae_l0.001 | 2048 | 0.001 | none | 44.61 | – | – | 53.02 | 48.3 |
| oae_l0.01/oae_sparse | 2048 | 0.01 | none | 42.94 ± 0.62 | 48.20 ± 0.14 | 50.65 ± 0.57 | 52.07 ± 0.42 | 49.72 |
| oae_sparse_denoise | 2048 | 0.01 | gaussian | 43.00 ± 0.61 | 48.59 ± 1.46 | 51.33 ± 0.41 | 52.85 ± 0.26 | 49.53 |
| oae_sparse_mask | 2048 | 0.01 | mask | 45.72 ± 0.18 | 49.67 ± 1.44 | 53.93 ± 0.09 | 55.66 ± 0.30 | 52.17 |
| oae_l0.1 | 2048 | 0.1 | none | 42.27 | – | – | 50.56 | 46.5 |
| oae | 2048 | 0.0 | none | 44.22 ± 0.91 | 49.39 ± 0.86 | 52.15 ± 0.20 | 52.88 ± 0.85 | 51.03 |
| oae_denoise | 2048 | 0.0 | gaussian | 44.13 ± 0.45 | 50.03 ± 0.24 | 52.55 ± 0.27 | 54.14 ± 0.10 | 50.67 |
| random | – | – | – | 35.58 ± 0.28 | 39.48 ± 0.34 | 41.61 ± 0.72 | 42.82 ± 1.13 | 41.8 |
## frozen — downstream training cost (mean over runs)
| canonical | train_time_s | peak_train_mem_mb | params_total | params_trainable |
|:---------------------|---------------:|--------------------:|---------------:|-------------------:|
| C128_l0.01_clean | 59.8 | 832.6 | 1.1174e+07 | 5130 |
| C128_l0.01_gauss0.5 | 57.3 | 832.6 | 1.1174e+07 | 5130 |
| C128_l0_clean | 56.6 | 832.6 | 1.1174e+07 | 5130 |
| C128_l0_gauss0.5 | 59.4 | 832.6 | 1.1174e+07 | 5130 |
| C2048_l0.001_clean | 64.2 | 832.6 | 1.1174e+07 | 5130 |
| C2048_l0.01_clean | 56.5 | 832.6 | 1.1174e+07 | 5130 |
| C2048_l0.01_gauss0.5 | 56.6 | 832.6 | 1.1174e+07 | 5130 |
| C2048_l0.01_mask0.5 | 57.9 | 832.6 | 1.1174e+07 | 5130 |
| C2048_l0.1_clean | 66.5 | 832.6 | 1.1174e+07 | 5130 |
| C2048_l0_clean | 61 | 832.6 | 1.1174e+07 | 5130 |
| C2048_l0_gauss0.5 | 56.5 | 832.6 | 1.1174e+07 | 5130 |
| random | 232.8 | 2418.4 | 1.1174e+07 | 5130 |
## frozen — paired comparisons (test top-1, B − A in points)
| A | B | fraction | n_seeds | diff_B_minus_A | diff_std | p_paired_t | p_paired_t_holm | cohens_dz | p_mcnemar_max_over_seeds |
|:------------------|:---------------------|-----------:|----------:|-----------------:|-----------:|-------------:|------------------:|------------:|---------------------------:|
| C128_l0_clean | C2048_l0.01_mask0.5 | 0.1 | 3 | 1.825 | 0.1146 | 0.001311 | 0.005244 | 15.93 | 0.0018 |
| C128_l0_clean | C2048_l0.01_mask0.5 | 0.25 | 3 | 1.133 | 2.788 | 0.5543 | 0.5543 | 0.4066 | 0.0002473 |
| C128_l0_clean | C2048_l0.01_mask0.5 | 0.5 | 3 | 3.029 | 0.5236 | 0.009812 | 0.02457 | 5.786 | 6.927e-06 |
| C128_l0_clean | C2048_l0.01_mask0.5 | 1 | 3 | 3.117 | 0.4915 | 0.00819 | 0.02457 | 6.341 | 9.169e-07 |
| C128_l0_clean | C2048_l0_clean | 0.1 | 3 | 0.3208 | 0.657 | 0.4867 | 1 | 0.4884 | 0.8107 |
| C128_l0_clean | C2048_l0_clean | 0.25 | 3 | 0.85 | 2.234 | 0.5776 | 1 | 0.3805 | 0.0009237 |
| C128_l0_clean | C2048_l0_clean | 0.5 | 3 | 1.242 | 0.5646 | 0.06253 | 0.2501 | 2.199 | 0.2492 |
| C128_l0_clean | C2048_l0_clean | 1 | 3 | 0.3333 | 0.7821 | 0.5372 | 1 | 0.4262 | 0.2995 |
| C2048_l0_clean | C2048_l0.01_clean | 0.1 | 3 | -1.279 | 0.8055 | 0.1107 | 0.332 | -1.588 | 0.4022 |
| C2048_l0_clean | C2048_l0.01_clean | 0.25 | 3 | -1.192 | 0.9306 | 0.1568 | 0.332 | -1.281 | 0.8221 |
| C2048_l0_clean | C2048_l0.01_clean | 0.5 | 3 | -1.496 | 0.6377 | 0.05557 | 0.2223 | -2.346 | 0.09288 |
| C2048_l0_clean | C2048_l0.01_clean | 1 | 3 | -0.8042 | 1.172 | 0.3567 | 0.3567 | -0.6861 | 0.3488 |
| C128_l0_clean | C128_l0.01_clean | 0.1 | 3 | 0.1125 | 0.5717 | 0.7657 | 1 | 0.1968 | 0.5233 |
| C128_l0_clean | C128_l0.01_clean | 0.25 | 3 | -0.2 | 1.61 | 0.8496 | 1 | -0.1242 | 0.2949 |
| C128_l0_clean | C128_l0.01_clean | 0.5 | 3 | -0.4958 | 0.6831 | 0.3356 | 1 | -0.7259 | 1 |
| C128_l0_clean | C128_l0.01_clean | 1 | 3 | -1.396 | 0.2306 | 0.008975 | 0.0359 | -6.053 | 0.01661 |
| C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.1 | 3 | 0.0625 | 0.598 | 0.873 | 1 | 0.1045 | 0.5002 |
| C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.25 | 3 | 0.3958 | 1.325 | 0.6564 | 1 | 0.2988 | 0.05433 |
| C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.5 | 3 | 0.675 | 0.3356 | 0.07345 | 0.2938 | 2.011 | 0.5933 |
| C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 1 | 3 | 0.7792 | 0.4661 | 0.1014 | 0.3043 | 1.672 | 0.5116 |
| C128_l0_clean | C128_l0_gauss0.5 | 0.1 | 3 | -1.008 | 0.593 | 0.09853 | 0.3941 | -1.7 | 0.5056 |
| C128_l0_clean | C128_l0_gauss0.5 | 0.25 | 3 | 0.1958 | 0.9284 | 0.7499 | 1 | 0.2109 | 0.203 |
| C128_l0_clean | C128_l0_gauss0.5 | 0.5 | 3 | 0.175 | 0.4265 | 0.5509 | 1 | 0.4103 | 0.9387 |
| C128_l0_clean | C128_l0_gauss0.5 | 1 | 3 | 0.1125 | 0.142 | 0.3036 | 0.9107 | 0.7924 | 0.9418 |
## Frozen vs fine-tuned (test_top1, mean over seeds)
| | finetune | frozen | finetune_minus_frozen |
|:-------------------------------|-----------:|---------:|------------------------:|
| ('C128_l0.01_clean', 0.1) | 47.9 | 44.01 | 3.9 |
| ('C128_l0.01_clean', 0.25) | 57.55 | 48.34 | 9.21 |
| ('C128_l0.01_clean', 0.5) | 64.12 | 50.41 | 13.71 |
| ('C128_l0.01_clean', 1.0) | 67.9 | 51.15 | 16.75 |
| ('C128_l0.01_gauss0.5', 0.1) | 48.93 | 42.26 | 6.67 |
| ('C128_l0.01_gauss0.5', 0.25) | 57.14 | 47.79 | 9.35 |
| ('C128_l0.01_gauss0.5', 0.5) | 63.78 | 49.28 | 14.5 |
| ('C128_l0.01_gauss0.5', 1.0) | 68.45 | 50.71 | 17.74 |
| ('C128_l0_clean', 0.1) | 49.19 | 43.9 | 5.3 |
| ('C128_l0_clean', 0.25) | 58.44 | 48.54 | 9.9 |
| ('C128_l0_clean', 0.5) | 65.8 | 50.9 | 14.9 |
| ('C128_l0_clean', 1.0) | 68.91 | 52.55 | 16.36 |
| ('C128_l0_gauss0.5', 0.1) | 48.88 | 42.89 | 5.99 |
| ('C128_l0_gauss0.5', 0.25) | 58.58 | 48.73 | 9.85 |
| ('C128_l0_gauss0.5', 0.5) | 65.35 | 51.08 | 14.27 |
| ('C128_l0_gauss0.5', 1.0) | 69.5 | 52.66 | 16.84 |
| ('C2048_l0.01_clean', 0.1) | 49.22 | 42.94 | 6.28 |
| ('C2048_l0.01_clean', 0.25) | 57.58 | 48.2 | 9.38 |
| ('C2048_l0.01_clean', 0.5) | 64.71 | 50.65 | 14.06 |
| ('C2048_l0.01_clean', 1.0) | 67.9 | 52.08 | 15.82 |
| ('C2048_l0.01_gauss0.5', 0.1) | 48.85 | 43 | 5.85 |
| ('C2048_l0.01_gauss0.5', 0.25) | 57.6 | 48.59 | 9 |
| ('C2048_l0.01_gauss0.5', 0.5) | 65.05 | 51.32 | 13.73 |
| ('C2048_l0.01_gauss0.5', 1.0) | 69.51 | 52.85 | 16.66 |
| ('C2048_l0.01_mask0.5', 0.1) | 49.55 | 45.72 | 3.83 |
| ('C2048_l0.01_mask0.5', 0.25) | 59.21 | 49.67 | 9.54 |
| ('C2048_l0.01_mask0.5', 0.5) | 67.4 | 53.93 | 13.47 |
| ('C2048_l0.01_mask0.5', 1.0) | 69.82 | 55.66 | 14.15 |
| ('C2048_l0_clean', 0.1) | 49.77 | 44.22 | 5.55 |
| ('C2048_l0_clean', 0.25) | 55.69 | 49.39 | 6.3 |
| ('C2048_l0_clean', 0.5) | 65.09 | 52.15 | 12.95 |
| ('C2048_l0_clean', 1.0) | 69.55 | 52.88 | 16.67 |
| ('C2048_l0_gauss0.5', 0.1) | 48.93 | 44.13 | 4.8 |
| ('C2048_l0_gauss0.5', 0.25) | 57.85 | 50.03 | 7.82 |
| ('C2048_l0_gauss0.5', 0.5) | 65.55 | 52.55 | 13.01 |
| ('C2048_l0_gauss0.5', 1.0) | 69.77 | 54.14 | 15.63 |
## Autoencoder pre-training
| variant | latent_dim | overcomplete_x | params_encoder | params_decoder | train_time_h | img_per_s | peak_mem_mb | holdout_psnr | frac_active |
|:-------------------|-------------:|-----------------:|-----------------:|-----------------:|---------------:|------------:|--------------:|---------------:|--------------:|
| oae_l0.001 | 73728 | 2.667 | 12219456 | 2617155 | 0.3322 | 818.7 | 2550 | 25.1 | 0.22 |
| oae_l0.01 | 73728 | 2.667 | 12219456 | 2617155 | 0.3115 | 873.3 | 2550 | 25.05 | 0.1399 |
| oae_l0.1 | 73728 | 2.667 | 12219456 | 2617155 | 0.3331 | 816.6 | 2550 | 24.79 | 0.05403 |
| ae | 4608 | 0.167 | 11234496 | 1634115 | 0.2988 | 910.4 | 2507 | 25.1 | 0.7798 |
| ae_sparse | 4608 | 0.167 | 11234496 | 1634115 | 0.317 | 858.2 | 2507 | 24.78 | 0.396 |
| ae_denoise | 4608 | 0.167 | 11234496 | 1634115 | 0.294 | 925.1 | 2674 | 24.74 | 0.7822 |
| ae_sparse_denoise | 4608 | 0.167 | 11234496 | 1634115 | 0.3021 | 900.3 | 2674 | 24.44 | 0.3752 |
| oae | 73728 | 2.667 | 12219456 | 2617155 | 0.313 | 869.1 | 2550 | 25.17 | 0.2507 |
| oae_sparse | 73728 | 2.667 | 12219456 | 2617155 | 0.3115 | 873.3 | 2550 | 25.05 | 0.1399 |
| oae_denoise | 73728 | 2.667 | 12219456 | 2617155 | 0.2901 | 937.7 | 2564 | 24.79 | 0.2423 |
| oae_sparse_denoise | 73728 | 2.667 | 12219456 | 2617155 | 0.2947 | 923.1 | 2564 | 24.73 | 0.1336 |
| oae_sparse_mask | 73728 | 2.667 | 12219456 | 2617155 | 0.2928 | 928.9 | 2564 | 24.01 | 0.08135 |
## Inference cost (identical downstream architecture for all models)
Parameters: 11,173,962
| | latency_ms_median | latency_ms_p90 | throughput_img_s | peak_gpu_mem_mb |
|:------|--------------------:|-----------------:|-------------------:|------------------:|
| bs1 | 3.06 | 3.21 | 326 | 76.4 |
| bs256 | 45 | 45.2 | 5.69e+03 | 442 |
## Selections (validation only)
```
{'finetune/ResNet-18 baseline': {'chosen': 'scratch', 'mean_val_top1': {'scratch': np.float64(0.5984999999999999)}}, 'finetune/Standard AE': {'chosen': 'C128_l0_clean', 'mean_val_top1': {'C128_l0_clean': np.float64(0.587)}}, 'finetune/Sparse OAE': {'chosen': 'C2048_l0.01_mask0.5', 'mean_val_top1': {'C2048_l0.01_clean': np.float64(0.5791666666666667), 'C2048_l0.01_gauss0.5': np.float64(0.585), 'C2048_l0.01_mask0.5': np.float64(0.6015)}}, 'frozen/Random frozen ResNet-18 (control)': {'chosen': 'random', 'mean_val_top1': {'random': np.float64(0.418)}}, 'frozen/Standard AE': {'chosen': 'C128_l0_clean', 'mean_val_top1': {'C128_l0_clean': np.float64(0.49733333333333346)}}, 'frozen/Sparse OAE': {'chosen': 'C2048_l0.01_mask0.5', 'mean_val_top1': {'C2048_l0.01_clean': np.float64(0.4971666666666666), 'C2048_l0.01_gauss0.5': np.float64(0.4953333333333334), 'C2048_l0.01_mask0.5': np.float64(0.5216666666666668)}}}
```