Buckets:
| tags: | |
| - arxiv:2510.08570 | |
| - reproduction | |
| # Linearizer Reproduction — arXiv:2510.08570 ("Who Said Neural Networks Aren't Linear?") | |
| Reduced-scale reproduction of the **core claims** of the Linearizer paper (Berman, Hallak, Shocher), | |
| scope chosen by the user: one-step flow matching on MNIST 32×32. Grounded in the official repo | |
| [`assafshocher/Linearizer`](https://github.com/assafshocher/Linearizer) @ commit `adfca2c`, imported | |
| verbatim; training/eval logic adapted from the repo's `one_step/train_one_step.py` / `test_one_step.py` | |
| by the driver [`run_repro.py`](run_repro.py). | |
| ## Configuration (reduced scale) | |
| | Item | Setting | | |
| |---|---| | |
| | Model | Official Linearizer: `InverseUnet` g (4 invertible blocks: ActNorm → 2 affine couplings w/ Song-UNet conditioners, model_channels=16 → invertible 1×1 conv), time-conditioned via gx(t=1)/gy(t=0) modes; rank-8 time-dependent LoRA core over dim 1024 — **9,932,800 params** | | |
| | Dataset | MNIST train split, 60k images, resized 32×32, [0,1] | | |
| | Loss | Official FM loss: induced-space MSE + noisy reconstruction terms (MSE on x0, LPIPS on x1/x1_pred), noise_level 0.3 | | |
| | Optimizer | Adam lr 1e-4, batch **128** (official 256 does not fit 24 GB), **30 epochs** (~14,070 steps) vs official 201 | | |
| | Hardware | 1× A10G (a10g-small), 0.82 it/s, 4.8 h training | | |
| ## Results | |
| All metrics computed by `mnist_fm_eval/results.json` on the 30-epoch checkpoint `lin_final.pth`. | |
| | Claim | Paper | This reproduction | Verdict | | |
| |---|---|---|---| | |
| | Linearity under induced ⊕/⊙ ops (Lemmas 1–2) | exact by construction | f(x1⊕x2) vs f(x1)⊕f(x2): **rel. err 2–7×10⁻⁶** (fp32 precision) across 8 addition + 3 scaling checks | ✅ CONFIRMED | | |
| | One-step ≡ multi-step sampling (Euler, T=100) | collapse to a single operator B | image **MSE 4.9×10⁻¹⁰ (PSNR 93.1 dB)**; latent collapse rel. err **6.6×10⁻⁶**; FID between the two sample sets **0.77** | ✅ CONFIRMED (exact) | | |
| | One-step ≡ multi-step (RK, T=100) | paper reports 100-vs-1 fidelity PSNR 32.4 / MSE 3.0×10⁻⁴ | **PSNR 32.0 dB / MSE 6.3×10⁻⁴** | ✅ matches paper — the official RK matrix-B is a cruder discretization than the iterative RK sampler (also flagged by third-party reproducers); Euler collapse is exact, RK collapse is approximate | | |
| | Inversion via Moore–Penrose pseudoinverse (B⁻¹) | exact encoding (Lemma 7) | roundtrip x1→x0→x̂1: **PSNR 76.4 dB**, LPIPS 1.2×10⁻⁷ | ✅ CONFIRMED | | |
| | g invertibility | required for induced spaces | roundtrip max abs err **3.1×10⁻⁴** per pixel | ✅ holds to training precision | | |
| | Generation quality | MNIST FID at 201 epochs, multi-step ~127 (CelebA numbers in main table) | **FID 43.1** (one-step, T=100, 10k samples vs 10k real test, pytorch-fid Inception) | partial: generated digits are recognizable (see `mnist_fm_eval/samples_one_step.png`), but 30-epoch training is far short of the paper's 201 — absolute FID not comparable | | |
| ### Honest deviations / findings | |
| - **T=1000 vs T=100 sampling**: one-step T=1000 gives FID 52.1 vs 43.1 at T=100 — the *opposite* direction from the paper's reported ~8-point improvement (1000→1 vs 100→1). Since the Euler collapse itself is exact (FID one-vs-multi = 0.77), this reflects the sampler trajectory of a model under-trained at 30 epochs, not a failure of the collapse. | |
| - The RK one-step/multi-step gap (PSNR 32) is an **implementation property of the official code** (its matrix-B RK omits the nested terms of Appendix F), not an error in our reproduction — and it exactly reproduces the paper's own reported one-vs-100 fidelity numbers. | |
| - Training used batch 128 instead of 256 (official config OOMs on 24 GB cards) and 30/201 epochs (budget). | |
| ## Artifacts | |
| - [`run_repro.py`](run_repro.py) — reproduction driver (uv-runnable, PEP 723 pinned deps) | |
| - [`mnist_fm/models/lin_final.pth`](mnist_fm/models/lin_final.pth) — trained checkpoint (30 epochs, whole-module torch.save) | |
| - [`mnist_fm_eval/results.json`](mnist_fm_eval/results.json) — all eval metrics | |
| - [`mnist_fm_eval/samples_one_step.png`](mnist_fm_eval/samples_one_step.png) / [`samples_multi_step.png`](mnist_fm_eval/samples_multi_step.png) — 24-sample grids (visually indistinguishable) | |
| - [`mnist_fm/artifacts/`](mnist_fm/artifacts/) — training-time sample grids every 5 epochs | |
| - [`mnist_fm/history.json`](mnist_fm/history.json) — per-epoch losses (0.42 → 0.126) | |
| - [`mnist_fm/code/Linearizer-official/`](mnist_fm/code/Linearizer-official/) — official repo snapshot @ `adfca2c` | |
| - [`mnist_fm/args.json`](mnist_fm/args.json) — exact run configuration | |
| ## Rerun | |
| ```bash | |
| uv run run_repro.py --epochs 30 --batch-size 128 --fid-samples 10000 \ | |
| --trackio-space <space_id> --out <out_dir> | |
| ``` | |
| Training dashboard: https://huggingface.co/spaces/abidlabs/linearizer-mnist-fm-trackio | |
| ## Provenance | |
| Trained on Hugging Face Jobs (a10g-small) with trackio logging; driver imports the official | |
| implementation verbatim from codeload tarball of commit `adfca2c`. arxiv:2510.08570 |
Xet Storage Details
- Size:
- 5.03 kB
- Xet hash:
- f8acce8b6d84b25057c56a3f99371d8da05b34f346d6dd3a2780d2b26e4d9d0e
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.