--- tags: - autoencoder - sparse-coding - representation-learning - self-supervised - stl10 - resnet18 --- # Overcomplete Sparse Autoencoders for Visual Representation Learning (STL-10) Controlled study of whether an **overcomplete, L1-sparse convolutional autoencoder** learns features that transfer better to STL-10 classification than a **conventional undercomplete autoencoder**, against a **supervised ResNet-18 trained from scratch**, at 10 / 25 / 50 / 100 % of the labelled data. Code: https://github.com/0x-khalil/OSAE * Shared feature extractor: ResNet-18 adapted to 96×96 (3×3/2 stem, no max-pool); identical downstream classifier for all models. * Standard AE latent: 128×6×6 = 4,608 coefficients (0.17× input). Overcomplete AE latent: 2048×6×6 = 73,728 (2.67× input). * Pre-training on the 100,000 unlabeled images; λ* = 0.01 (selected on validation). * Downstream: full fine-tuning and frozen-encoder protocols, 3 seeds, identical stratified subsets, validation-only model selection, test split used once for final evaluation. ## Files | Path | Content | |---|---| | `results/RESULTS.md` | headline table, ablations, paired statistics, costs | | `results/figures/` | label-efficiency curves, confusion matrices, per-class accuracy | | `results/representation/` | sparsity, reconstruction, separability, PCA / t-SNE / UMAP, CKA | | `encoders//encoder.pt` | pre-trained ResNet-18 trunk (`torch.load(...)["encoder"]`) | | `downstream/` | per-run metrics (and checkpoints if uploaded) | | `configs/`, `data_splits/` | exact configuration and split indices | Load a pre-trained trunk with the code from the repository: ```python import torch from osae.models import Classifier clf = Classifier() clf.encoder.load_state_dict(torch.load("encoders/oae_sparse/encoder.pt")["encoder"]) ``` --- # Results All numbers: mean ± std over seeds, in %. Test = official STL-10 test split (8,000 images); every choice below was made on the validation split. ## finetune — test_top1 | Model | variant | 10% | 25% | 50% | 100% | |:-------------------|:--------------------|:-------------|:-------------|:-------------|:-------------| | ResNet-18 baseline | scratch | 44.11 ± 0.62 | 54.44 ± 1.86 | 65.16 ± 0.97 | 72.77 ± 0.40 | | Standard AE | C128_l0_clean | 49.19 ± 0.28 | 58.44 ± 0.29 | 65.80 ± 0.12 | 68.91 ± 0.64 | | Sparse OAE | C2048_l0.01_mask0.5 | 49.55 ± 0.84 | 59.21 ± 0.54 | 67.40 ± 0.14 | 69.82 ± 4.37 | ## finetune — test_macro_f1 | Model | variant | 10% | 25% | 50% | 100% | |:-------------------|:--------------------|:-------------|:-------------|:-------------|:-------------| | ResNet-18 baseline | scratch | 44.00 ± 0.65 | 54.40 ± 1.93 | 65.21 ± 0.97 | 72.87 ± 0.39 | | Standard AE | C128_l0_clean | 49.02 ± 0.28 | 58.46 ± 0.28 | 65.84 ± 0.16 | 68.81 ± 0.77 | | Sparse OAE | C2048_l0.01_mask0.5 | 49.36 ± 0.82 | 59.16 ± 0.46 | 67.45 ± 0.15 | 69.76 ± 4.46 | ## finetune — val_top1 | Model | variant | 10% | 25% | 50% | 100% | |:-------------------|:--------------------|:-------------|:-------------|:-------------|:-------------| | ResNet-18 baseline | scratch | 47.47 ± 2.31 | 57.20 ± 1.22 | 64.40 ± 1.59 | 70.33 ± 1.10 | | Standard AE | C128_l0_clean | 51.13 ± 0.42 | 57.07 ± 0.95 | 62.27 ± 1.50 | 64.33 ± 1.30 | | Sparse OAE | C2048_l0.01_mask0.5 | 50.40 ± 1.83 | 59.33 ± 0.31 | 64.07 ± 0.70 | 66.80 ± 0.53 | ## finetune — all variants (test_top1) | variant | latent | lambda | noise | 10% | 25% | 50% | 100% | val (all f) | |:---------------------|:---------|:---------|:---------|:-------------|:-------------|:-------------|:-------------|--------------:| | ae_sparse | 128 | 0.01 | none | 47.90 ± 0.87 | 57.55 ± 0.64 | 64.12 ± 0.63 | 67.90 ± 0.79 | 59.02 | | ae_sparse_denoise | 128 | 0.01 | gaussian | 48.93 ± 0.15 | 57.14 ± 0.50 | 63.78 ± 0.94 | 68.45 ± 0.51 | 59 | | ae | 128 | 0.0 | none | 49.19 ± 0.28 | 58.44 ± 0.29 | 65.80 ± 0.12 | 68.91 ± 0.64 | 58.7 | | ae_denoise | 128 | 0.0 | gaussian | 48.88 ± 0.48 | 58.58 ± 0.24 | 65.35 ± 0.73 | 69.50 ± 0.35 | 59.17 | | oae_l0.01/oae_sparse | 2048 | 0.01 | none | 49.22 ± 0.51 | 57.58 ± 0.64 | 64.71 ± 0.39 | 67.90 ± 0.99 | 57.92 | | oae_sparse_denoise | 2048 | 0.01 | gaussian | 48.85 ± 0.66 | 57.60 ± 0.53 | 65.05 ± 0.52 | 69.51 ± 0.14 | 58.5 | | oae_sparse_mask | 2048 | 0.01 | mask | 49.55 ± 0.84 | 59.21 ± 0.54 | 67.40 ± 0.14 | 69.82 ± 4.37 | 60.15 | | oae | 2048 | 0.0 | none | 49.77 ± 0.46 | 55.69 ± 1.58 | 65.09 ± 0.33 | 69.55 ± 0.48 | 58.33 | | oae_denoise | 2048 | 0.0 | gaussian | 48.93 ± 0.40 | 57.85 ± 0.46 | 65.55 ± 0.16 | 69.77 ± 0.21 | 59.35 | | scratch | – | – | – | 44.11 ± 0.62 | 54.44 ± 1.86 | 65.16 ± 0.97 | 72.77 ± 0.40 | 59.85 | ## finetune — downstream training cost (mean over runs) | canonical | train_time_s | peak_train_mem_mb | params_total | params_trainable | |:---------------------|---------------:|--------------------:|---------------:|-------------------:| | C128_l0.01_clean | 147.5 | 919.2 | 1.1174e+07 | 1.1174e+07 | | C128_l0.01_gauss0.5 | 145.1 | 919.2 | 1.1174e+07 | 1.1174e+07 | | C128_l0_clean | 153 | 919.2 | 1.1174e+07 | 1.1174e+07 | | C128_l0_gauss0.5 | 149.9 | 919.2 | 1.1174e+07 | 1.1174e+07 | | C2048_l0.01_clean | 148.4 | 919.2 | 1.1174e+07 | 1.1174e+07 | | C2048_l0.01_gauss0.5 | 142 | 919.2 | 1.1174e+07 | 1.1174e+07 | | C2048_l0.01_mask0.5 | 143.9 | 919.2 | 1.1174e+07 | 1.1174e+07 | | C2048_l0_clean | 148.7 | 919.2 | 1.1174e+07 | 1.1174e+07 | | C2048_l0_gauss0.5 | 147.4 | 919.2 | 1.1174e+07 | 1.1174e+07 | | scratch | 151 | 919.2 | 1.1174e+07 | 1.1174e+07 | ## finetune — paired comparisons (test top-1, B − A in points) | A | B | fraction | n_seeds | diff_B_minus_A | diff_std | p_paired_t | p_paired_t_holm | cohens_dz | p_mcnemar_max_over_seeds | |:------------------|:---------------------|-----------:|----------:|-----------------:|-----------:|-------------:|------------------:|------------:|---------------------------:| | scratch | C128_l0_clean | 0.1 | 3 | 5.083 | 0.719 | 0.006602 | 0.02641 | 7.07 | 2.226e-14 | | scratch | C128_l0_clean | 0.25 | 3 | 4.004 | 2.073 | 0.07893 | 0.1579 | 1.931 | 0.002828 | | scratch | C128_l0_clean | 0.5 | 3 | 0.6417 | 1.079 | 0.4112 | 0.4112 | 0.5947 | 0.3509 | | scratch | C128_l0_clean | 1 | 3 | -3.858 | 0.5837 | 0.007542 | 0.02641 | -6.61 | 7.597e-11 | | scratch | C2048_l0.01_mask0.5 | 0.1 | 3 | 5.438 | 0.9135 | 0.009278 | 0.03711 | 5.952 | 6.667e-17 | | scratch | C2048_l0.01_mask0.5 | 0.25 | 3 | 4.775 | 1.338 | 0.0252 | 0.07559 | 3.568 | 2.118e-09 | | scratch | C2048_l0.01_mask0.5 | 0.5 | 3 | 2.237 | 1.038 | 0.06488 | 0.1298 | 2.155 | 0.01869 | | scratch | C2048_l0.01_mask0.5 | 1 | 3 | -2.95 | 4.179 | 0.346 | 0.346 | -0.7059 | 0.5993 | | C128_l0_clean | C2048_l0.01_mask0.5 | 0.1 | 3 | 0.3542 | 1.122 | 0.6394 | 1 | 0.3157 | 0.3909 | | C128_l0_clean | C2048_l0.01_mask0.5 | 0.25 | 3 | 0.7708 | 0.8079 | 0.2402 | 0.7207 | 0.9541 | 0.7619 | | C128_l0_clean | C2048_l0.01_mask0.5 | 0.5 | 3 | 1.596 | 0.1938 | 0.004879 | 0.01952 | 8.235 | 0.004674 | | C128_l0_clean | C2048_l0.01_mask0.5 | 1 | 3 | 0.9083 | 3.722 | 0.7136 | 1 | 0.244 | 4.28e-09 | | C128_l0_clean | C2048_l0_clean | 0.1 | 3 | 0.5792 | 0.6411 | 0.2581 | 0.3788 | 0.9034 | 0.8093 | | C128_l0_clean | C2048_l0_clean | 0.25 | 3 | -2.75 | 1.875 | 0.1263 | 0.3788 | -1.467 | 0.05449 | | C128_l0_clean | C2048_l0_clean | 0.5 | 3 | -0.7125 | 0.219 | 0.03008 | 0.1203 | -3.253 | 0.2517 | | C128_l0_clean | C2048_l0_clean | 1 | 3 | 0.6375 | 0.553 | 0.1839 | 0.3788 | 1.153 | 1 | | C2048_l0_clean | C2048_l0.01_clean | 0.1 | 3 | -0.55 | 0.2382 | 0.05719 | 0.2288 | -2.309 | 0.3679 | | C2048_l0_clean | C2048_l0.01_clean | 0.25 | 3 | 1.887 | 1.003 | 0.08257 | 0.2477 | 1.883 | 0.01547 | | C2048_l0_clean | C2048_l0.01_clean | 0.5 | 3 | -0.3792 | 0.7142 | 0.4549 | 0.4549 | -0.5309 | 0.3409 | | C2048_l0_clean | C2048_l0.01_clean | 1 | 3 | -1.646 | 1.06 | 0.1149 | 0.2477 | -1.553 | 0.1841 | | C128_l0_clean | C128_l0.01_clean | 0.1 | 3 | -1.288 | 0.7253 | 0.0915 | 0.2745 | -1.775 | 0.1862 | | C128_l0_clean | C128_l0.01_clean | 0.25 | 3 | -0.8917 | 0.7381 | 0.1715 | 0.343 | -1.208 | 0.7835 | | C128_l0_clean | C128_l0.01_clean | 0.5 | 3 | -1.683 | 0.5693 | 0.03608 | 0.1443 | -2.957 | 0.01671 | | C128_l0_clean | C128_l0.01_clean | 1 | 3 | -1.008 | 1.264 | 0.3012 | 0.343 | -0.7977 | 0.4219 | | C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.1 | 3 | -0.3708 | 0.4537 | 0.2925 | 0.8775 | -0.8174 | 0.824 | | C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.25 | 3 | 0.01667 | 1.096 | 0.9814 | 0.9814 | 0.01521 | 0.2753 | | C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.5 | 3 | 0.3417 | 0.6825 | 0.4773 | 0.9546 | 0.5006 | 0.6171 | | C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 1 | 3 | 1.613 | 0.8928 | 0.08878 | 0.3551 | 1.806 | 0.02627 | | C128_l0_clean | C128_l0_gauss0.5 | 0.1 | 3 | -0.3167 | 0.3224 | 0.231 | 0.9241 | -0.9821 | 0.9317 | | C128_l0_clean | C128_l0_gauss0.5 | 0.25 | 3 | 0.1417 | 0.4161 | 0.6152 | 0.9852 | 0.3404 | 0.4568 | | C128_l0_clean | C128_l0_gauss0.5 | 0.5 | 3 | -0.4583 | 0.6193 | 0.3284 | 0.9852 | -0.7401 | 0.6854 | | C128_l0_clean | C128_l0_gauss0.5 | 1 | 3 | 0.5917 | 0.9863 | 0.4079 | 0.9852 | 0.5999 | 0.5875 | ## frozen — test_top1 | Model | variant | 10% | 25% | 50% | 100% | |:----------------------------------|:--------------------|:-------------|:-------------|:-------------|:-------------| | Random frozen ResNet-18 (control) | random | 35.58 ± 0.28 | 39.48 ± 0.34 | 41.61 ± 0.72 | 42.82 ± 1.13 | | Standard AE | C128_l0_clean | 43.90 ± 0.29 | 48.54 ± 1.39 | 50.90 ± 0.56 | 52.55 ± 0.21 | | Sparse OAE | C2048_l0.01_mask0.5 | 45.72 ± 0.18 | 49.67 ± 1.44 | 53.93 ± 0.09 | 55.66 ± 0.30 | ## frozen — test_macro_f1 | Model | variant | 10% | 25% | 50% | 100% | |:----------------------------------|:--------------------|:-------------|:-------------|:-------------|:-------------| | Random frozen ResNet-18 (control) | random | 35.26 ± 0.24 | 39.22 ± 0.38 | 41.17 ± 0.66 | 42.36 ± 1.14 | | Standard AE | C128_l0_clean | 43.83 ± 0.30 | 48.31 ± 1.34 | 50.68 ± 0.62 | 52.26 ± 0.26 | | Sparse OAE | C2048_l0.01_mask0.5 | 45.50 ± 0.19 | 49.52 ± 1.48 | 53.86 ± 0.08 | 55.54 ± 0.33 | ## frozen — val_top1 | Model | variant | 10% | 25% | 50% | 100% | |:----------------------------------|:--------------------|:-------------|:-------------|:-------------|:-------------| | Random frozen ResNet-18 (control) | random | 37.20 ± 1.56 | 41.93 ± 1.01 | 42.80 ± 3.33 | 45.27 ± 2.34 | | Standard AE | C128_l0_clean | 45.60 ± 0.20 | 50.27 ± 0.81 | 50.73 ± 1.14 | 52.33 ± 1.21 | | Sparse OAE | C2048_l0.01_mask0.5 | 46.33 ± 0.50 | 51.20 ± 0.20 | 54.53 ± 0.83 | 56.60 ± 0.53 | ## frozen — all variants (test_top1) | variant | latent | lambda | noise | 10% | 25% | 50% | 100% | val (all f) | |:---------------------|:---------|:---------|:---------|:-------------|:-------------|:-------------|:-------------|--------------:| | ae_sparse | 128 | 0.01 | none | 44.01 ± 0.35 | 48.34 ± 0.25 | 50.41 ± 0.13 | 51.15 ± 0.36 | 49 | | ae_sparse_denoise | 128 | 0.01 | gaussian | 42.26 ± 0.89 | 47.79 ± 0.43 | 49.28 ± 0.30 | 50.71 ± 0.29 | 47.5 | | ae | 128 | 0.0 | none | 43.90 ± 0.29 | 48.54 ± 1.39 | 50.90 ± 0.56 | 52.55 ± 0.21 | 49.73 | | ae_denoise | 128 | 0.0 | gaussian | 42.89 ± 0.56 | 48.73 ± 0.47 | 51.08 ± 0.47 | 52.66 ± 0.31 | 48.43 | | oae_l0.001 | 2048 | 0.001 | none | 44.61 | – | – | 53.02 | 48.3 | | oae_l0.01/oae_sparse | 2048 | 0.01 | none | 42.94 ± 0.62 | 48.20 ± 0.14 | 50.65 ± 0.57 | 52.07 ± 0.42 | 49.72 | | oae_sparse_denoise | 2048 | 0.01 | gaussian | 43.00 ± 0.61 | 48.59 ± 1.46 | 51.33 ± 0.41 | 52.85 ± 0.26 | 49.53 | | oae_sparse_mask | 2048 | 0.01 | mask | 45.72 ± 0.18 | 49.67 ± 1.44 | 53.93 ± 0.09 | 55.66 ± 0.30 | 52.17 | | oae_l0.1 | 2048 | 0.1 | none | 42.27 | – | – | 50.56 | 46.5 | | oae | 2048 | 0.0 | none | 44.22 ± 0.91 | 49.39 ± 0.86 | 52.15 ± 0.20 | 52.88 ± 0.85 | 51.03 | | oae_denoise | 2048 | 0.0 | gaussian | 44.13 ± 0.45 | 50.03 ± 0.24 | 52.55 ± 0.27 | 54.14 ± 0.10 | 50.67 | | random | – | – | – | 35.58 ± 0.28 | 39.48 ± 0.34 | 41.61 ± 0.72 | 42.82 ± 1.13 | 41.8 | ## frozen — downstream training cost (mean over runs) | canonical | train_time_s | peak_train_mem_mb | params_total | params_trainable | |:---------------------|---------------:|--------------------:|---------------:|-------------------:| | C128_l0.01_clean | 59.8 | 832.6 | 1.1174e+07 | 5130 | | C128_l0.01_gauss0.5 | 57.3 | 832.6 | 1.1174e+07 | 5130 | | C128_l0_clean | 56.6 | 832.6 | 1.1174e+07 | 5130 | | C128_l0_gauss0.5 | 59.4 | 832.6 | 1.1174e+07 | 5130 | | C2048_l0.001_clean | 64.2 | 832.6 | 1.1174e+07 | 5130 | | C2048_l0.01_clean | 56.5 | 832.6 | 1.1174e+07 | 5130 | | C2048_l0.01_gauss0.5 | 56.6 | 832.6 | 1.1174e+07 | 5130 | | C2048_l0.01_mask0.5 | 57.9 | 832.6 | 1.1174e+07 | 5130 | | C2048_l0.1_clean | 66.5 | 832.6 | 1.1174e+07 | 5130 | | C2048_l0_clean | 61 | 832.6 | 1.1174e+07 | 5130 | | C2048_l0_gauss0.5 | 56.5 | 832.6 | 1.1174e+07 | 5130 | | random | 232.8 | 2418.4 | 1.1174e+07 | 5130 | ## frozen — paired comparisons (test top-1, B − A in points) | A | B | fraction | n_seeds | diff_B_minus_A | diff_std | p_paired_t | p_paired_t_holm | cohens_dz | p_mcnemar_max_over_seeds | |:------------------|:---------------------|-----------:|----------:|-----------------:|-----------:|-------------:|------------------:|------------:|---------------------------:| | C128_l0_clean | C2048_l0.01_mask0.5 | 0.1 | 3 | 1.825 | 0.1146 | 0.001311 | 0.005244 | 15.93 | 0.0018 | | C128_l0_clean | C2048_l0.01_mask0.5 | 0.25 | 3 | 1.133 | 2.788 | 0.5543 | 0.5543 | 0.4066 | 0.0002473 | | C128_l0_clean | C2048_l0.01_mask0.5 | 0.5 | 3 | 3.029 | 0.5236 | 0.009812 | 0.02457 | 5.786 | 6.927e-06 | | C128_l0_clean | C2048_l0.01_mask0.5 | 1 | 3 | 3.117 | 0.4915 | 0.00819 | 0.02457 | 6.341 | 9.169e-07 | | C128_l0_clean | C2048_l0_clean | 0.1 | 3 | 0.3208 | 0.657 | 0.4867 | 1 | 0.4884 | 0.8107 | | C128_l0_clean | C2048_l0_clean | 0.25 | 3 | 0.85 | 2.234 | 0.5776 | 1 | 0.3805 | 0.0009237 | | C128_l0_clean | C2048_l0_clean | 0.5 | 3 | 1.242 | 0.5646 | 0.06253 | 0.2501 | 2.199 | 0.2492 | | C128_l0_clean | C2048_l0_clean | 1 | 3 | 0.3333 | 0.7821 | 0.5372 | 1 | 0.4262 | 0.2995 | | C2048_l0_clean | C2048_l0.01_clean | 0.1 | 3 | -1.279 | 0.8055 | 0.1107 | 0.332 | -1.588 | 0.4022 | | C2048_l0_clean | C2048_l0.01_clean | 0.25 | 3 | -1.192 | 0.9306 | 0.1568 | 0.332 | -1.281 | 0.8221 | | C2048_l0_clean | C2048_l0.01_clean | 0.5 | 3 | -1.496 | 0.6377 | 0.05557 | 0.2223 | -2.346 | 0.09288 | | C2048_l0_clean | C2048_l0.01_clean | 1 | 3 | -0.8042 | 1.172 | 0.3567 | 0.3567 | -0.6861 | 0.3488 | | C128_l0_clean | C128_l0.01_clean | 0.1 | 3 | 0.1125 | 0.5717 | 0.7657 | 1 | 0.1968 | 0.5233 | | C128_l0_clean | C128_l0.01_clean | 0.25 | 3 | -0.2 | 1.61 | 0.8496 | 1 | -0.1242 | 0.2949 | | C128_l0_clean | C128_l0.01_clean | 0.5 | 3 | -0.4958 | 0.6831 | 0.3356 | 1 | -0.7259 | 1 | | C128_l0_clean | C128_l0.01_clean | 1 | 3 | -1.396 | 0.2306 | 0.008975 | 0.0359 | -6.053 | 0.01661 | | C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.1 | 3 | 0.0625 | 0.598 | 0.873 | 1 | 0.1045 | 0.5002 | | C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.25 | 3 | 0.3958 | 1.325 | 0.6564 | 1 | 0.2988 | 0.05433 | | C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 0.5 | 3 | 0.675 | 0.3356 | 0.07345 | 0.2938 | 2.011 | 0.5933 | | C2048_l0.01_clean | C2048_l0.01_gauss0.5 | 1 | 3 | 0.7792 | 0.4661 | 0.1014 | 0.3043 | 1.672 | 0.5116 | | C128_l0_clean | C128_l0_gauss0.5 | 0.1 | 3 | -1.008 | 0.593 | 0.09853 | 0.3941 | -1.7 | 0.5056 | | C128_l0_clean | C128_l0_gauss0.5 | 0.25 | 3 | 0.1958 | 0.9284 | 0.7499 | 1 | 0.2109 | 0.203 | | C128_l0_clean | C128_l0_gauss0.5 | 0.5 | 3 | 0.175 | 0.4265 | 0.5509 | 1 | 0.4103 | 0.9387 | | C128_l0_clean | C128_l0_gauss0.5 | 1 | 3 | 0.1125 | 0.142 | 0.3036 | 0.9107 | 0.7924 | 0.9418 | ## Frozen vs fine-tuned (test_top1, mean over seeds) | | finetune | frozen | finetune_minus_frozen | |:-------------------------------|-----------:|---------:|------------------------:| | ('C128_l0.01_clean', 0.1) | 47.9 | 44.01 | 3.9 | | ('C128_l0.01_clean', 0.25) | 57.55 | 48.34 | 9.21 | | ('C128_l0.01_clean', 0.5) | 64.12 | 50.41 | 13.71 | | ('C128_l0.01_clean', 1.0) | 67.9 | 51.15 | 16.75 | | ('C128_l0.01_gauss0.5', 0.1) | 48.93 | 42.26 | 6.67 | | ('C128_l0.01_gauss0.5', 0.25) | 57.14 | 47.79 | 9.35 | | ('C128_l0.01_gauss0.5', 0.5) | 63.78 | 49.28 | 14.5 | | ('C128_l0.01_gauss0.5', 1.0) | 68.45 | 50.71 | 17.74 | | ('C128_l0_clean', 0.1) | 49.19 | 43.9 | 5.3 | | ('C128_l0_clean', 0.25) | 58.44 | 48.54 | 9.9 | | ('C128_l0_clean', 0.5) | 65.8 | 50.9 | 14.9 | | ('C128_l0_clean', 1.0) | 68.91 | 52.55 | 16.36 | | ('C128_l0_gauss0.5', 0.1) | 48.88 | 42.89 | 5.99 | | ('C128_l0_gauss0.5', 0.25) | 58.58 | 48.73 | 9.85 | | ('C128_l0_gauss0.5', 0.5) | 65.35 | 51.08 | 14.27 | | ('C128_l0_gauss0.5', 1.0) | 69.5 | 52.66 | 16.84 | | ('C2048_l0.01_clean', 0.1) | 49.22 | 42.94 | 6.28 | | ('C2048_l0.01_clean', 0.25) | 57.58 | 48.2 | 9.38 | | ('C2048_l0.01_clean', 0.5) | 64.71 | 50.65 | 14.06 | | ('C2048_l0.01_clean', 1.0) | 67.9 | 52.08 | 15.82 | | ('C2048_l0.01_gauss0.5', 0.1) | 48.85 | 43 | 5.85 | | ('C2048_l0.01_gauss0.5', 0.25) | 57.6 | 48.59 | 9 | | ('C2048_l0.01_gauss0.5', 0.5) | 65.05 | 51.32 | 13.73 | | ('C2048_l0.01_gauss0.5', 1.0) | 69.51 | 52.85 | 16.66 | | ('C2048_l0.01_mask0.5', 0.1) | 49.55 | 45.72 | 3.83 | | ('C2048_l0.01_mask0.5', 0.25) | 59.21 | 49.67 | 9.54 | | ('C2048_l0.01_mask0.5', 0.5) | 67.4 | 53.93 | 13.47 | | ('C2048_l0.01_mask0.5', 1.0) | 69.82 | 55.66 | 14.15 | | ('C2048_l0_clean', 0.1) | 49.77 | 44.22 | 5.55 | | ('C2048_l0_clean', 0.25) | 55.69 | 49.39 | 6.3 | | ('C2048_l0_clean', 0.5) | 65.09 | 52.15 | 12.95 | | ('C2048_l0_clean', 1.0) | 69.55 | 52.88 | 16.67 | | ('C2048_l0_gauss0.5', 0.1) | 48.93 | 44.13 | 4.8 | | ('C2048_l0_gauss0.5', 0.25) | 57.85 | 50.03 | 7.82 | | ('C2048_l0_gauss0.5', 0.5) | 65.55 | 52.55 | 13.01 | | ('C2048_l0_gauss0.5', 1.0) | 69.77 | 54.14 | 15.63 | ## Autoencoder pre-training | variant | latent_dim | overcomplete_x | params_encoder | params_decoder | train_time_h | img_per_s | peak_mem_mb | holdout_psnr | frac_active | |:-------------------|-------------:|-----------------:|-----------------:|-----------------:|---------------:|------------:|--------------:|---------------:|--------------:| | oae_l0.001 | 73728 | 2.667 | 12219456 | 2617155 | 0.3322 | 818.7 | 2550 | 25.1 | 0.22 | | oae_l0.01 | 73728 | 2.667 | 12219456 | 2617155 | 0.3115 | 873.3 | 2550 | 25.05 | 0.1399 | | oae_l0.1 | 73728 | 2.667 | 12219456 | 2617155 | 0.3331 | 816.6 | 2550 | 24.79 | 0.05403 | | ae | 4608 | 0.167 | 11234496 | 1634115 | 0.2988 | 910.4 | 2507 | 25.1 | 0.7798 | | ae_sparse | 4608 | 0.167 | 11234496 | 1634115 | 0.317 | 858.2 | 2507 | 24.78 | 0.396 | | ae_denoise | 4608 | 0.167 | 11234496 | 1634115 | 0.294 | 925.1 | 2674 | 24.74 | 0.7822 | | ae_sparse_denoise | 4608 | 0.167 | 11234496 | 1634115 | 0.3021 | 900.3 | 2674 | 24.44 | 0.3752 | | oae | 73728 | 2.667 | 12219456 | 2617155 | 0.313 | 869.1 | 2550 | 25.17 | 0.2507 | | oae_sparse | 73728 | 2.667 | 12219456 | 2617155 | 0.3115 | 873.3 | 2550 | 25.05 | 0.1399 | | oae_denoise | 73728 | 2.667 | 12219456 | 2617155 | 0.2901 | 937.7 | 2564 | 24.79 | 0.2423 | | oae_sparse_denoise | 73728 | 2.667 | 12219456 | 2617155 | 0.2947 | 923.1 | 2564 | 24.73 | 0.1336 | | oae_sparse_mask | 73728 | 2.667 | 12219456 | 2617155 | 0.2928 | 928.9 | 2564 | 24.01 | 0.08135 | ## Inference cost (identical downstream architecture for all models) Parameters: 11,173,962 | | latency_ms_median | latency_ms_p90 | throughput_img_s | peak_gpu_mem_mb | |:------|--------------------:|-----------------:|-------------------:|------------------:| | bs1 | 3.06 | 3.21 | 326 | 76.4 | | bs256 | 45 | 45.2 | 5.69e+03 | 442 | ## Selections (validation only) ``` {'finetune/ResNet-18 baseline': {'chosen': 'scratch', 'mean_val_top1': {'scratch': np.float64(0.5984999999999999)}}, 'finetune/Standard AE': {'chosen': 'C128_l0_clean', 'mean_val_top1': {'C128_l0_clean': np.float64(0.587)}}, 'finetune/Sparse OAE': {'chosen': 'C2048_l0.01_mask0.5', 'mean_val_top1': {'C2048_l0.01_clean': np.float64(0.5791666666666667), 'C2048_l0.01_gauss0.5': np.float64(0.585), 'C2048_l0.01_mask0.5': np.float64(0.6015)}}, 'frozen/Random frozen ResNet-18 (control)': {'chosen': 'random', 'mean_val_top1': {'random': np.float64(0.418)}}, 'frozen/Standard AE': {'chosen': 'C128_l0_clean', 'mean_val_top1': {'C128_l0_clean': np.float64(0.49733333333333346)}}, 'frozen/Sparse OAE': {'chosen': 'C2048_l0.01_mask0.5', 'mean_val_top1': {'C2048_l0.01_clean': np.float64(0.4971666666666666), 'C2048_l0.01_gauss0.5': np.float64(0.4953333333333334), 'C2048_l0.01_mask0.5': np.float64(0.5216666666666668)}}} ```