abidlabs's picture
|
download
raw
5.03 kB
---
tags:
- arxiv:2510.08570
- reproduction
---
# Linearizer Reproduction — arXiv:2510.08570 ("Who Said Neural Networks Aren't Linear?")
Reduced-scale reproduction of the **core claims** of the Linearizer paper (Berman, Hallak, Shocher),
scope chosen by the user: one-step flow matching on MNIST 32×32. Grounded in the official repo
[`assafshocher/Linearizer`](https://github.com/assafshocher/Linearizer) @ commit `adfca2c`, imported
verbatim; training/eval logic adapted from the repo's `one_step/train_one_step.py` / `test_one_step.py`
by the driver [`run_repro.py`](run_repro.py).
## Configuration (reduced scale)
| Item | Setting |
|---|---|
| Model | Official Linearizer: `InverseUnet` g (4 invertible blocks: ActNorm → 2 affine couplings w/ Song-UNet conditioners, model_channels=16 → invertible 1×1 conv), time-conditioned via gx(t=1)/gy(t=0) modes; rank-8 time-dependent LoRA core over dim 1024 — **9,932,800 params** |
| Dataset | MNIST train split, 60k images, resized 32×32, [0,1] |
| Loss | Official FM loss: induced-space MSE + noisy reconstruction terms (MSE on x0, LPIPS on x1/x1_pred), noise_level 0.3 |
| Optimizer | Adam lr 1e-4, batch **128** (official 256 does not fit 24 GB), **30 epochs** (~14,070 steps) vs official 201 |
| Hardware | 1× A10G (a10g-small), 0.82 it/s, 4.8 h training |
## Results
All metrics computed by `mnist_fm_eval/results.json` on the 30-epoch checkpoint `lin_final.pth`.
| Claim | Paper | This reproduction | Verdict |
|---|---|---|---|
| Linearity under induced ⊕/⊙ ops (Lemmas 1–2) | exact by construction | f(x1⊕x2) vs f(x1)⊕f(x2): **rel. err 2–7×10⁻⁶** (fp32 precision) across 8 addition + 3 scaling checks | ✅ CONFIRMED |
| One-step ≡ multi-step sampling (Euler, T=100) | collapse to a single operator B | image **MSE 4.9×10⁻¹⁰ (PSNR 93.1 dB)**; latent collapse rel. err **6.6×10⁻⁶**; FID between the two sample sets **0.77** | ✅ CONFIRMED (exact) |
| One-step ≡ multi-step (RK, T=100) | paper reports 100-vs-1 fidelity PSNR 32.4 / MSE 3.0×10⁻⁴ | **PSNR 32.0 dB / MSE 6.3×10⁻⁴** | ✅ matches paper — the official RK matrix-B is a cruder discretization than the iterative RK sampler (also flagged by third-party reproducers); Euler collapse is exact, RK collapse is approximate |
| Inversion via Moore–Penrose pseudoinverse (B⁻¹) | exact encoding (Lemma 7) | roundtrip x1→x0→x̂1: **PSNR 76.4 dB**, LPIPS 1.2×10⁻⁷ | ✅ CONFIRMED |
| g invertibility | required for induced spaces | roundtrip max abs err **3.1×10⁻⁴** per pixel | ✅ holds to training precision |
| Generation quality | MNIST FID at 201 epochs, multi-step ~127 (CelebA numbers in main table) | **FID 43.1** (one-step, T=100, 10k samples vs 10k real test, pytorch-fid Inception) | partial: generated digits are recognizable (see `mnist_fm_eval/samples_one_step.png`), but 30-epoch training is far short of the paper's 201 — absolute FID not comparable |
### Honest deviations / findings
- **T=1000 vs T=100 sampling**: one-step T=1000 gives FID 52.1 vs 43.1 at T=100 — the *opposite* direction from the paper's reported ~8-point improvement (1000→1 vs 100→1). Since the Euler collapse itself is exact (FID one-vs-multi = 0.77), this reflects the sampler trajectory of a model under-trained at 30 epochs, not a failure of the collapse.
- The RK one-step/multi-step gap (PSNR 32) is an **implementation property of the official code** (its matrix-B RK omits the nested terms of Appendix F), not an error in our reproduction — and it exactly reproduces the paper's own reported one-vs-100 fidelity numbers.
- Training used batch 128 instead of 256 (official config OOMs on 24 GB cards) and 30/201 epochs (budget).
## Artifacts
- [`run_repro.py`](run_repro.py) — reproduction driver (uv-runnable, PEP 723 pinned deps)
- [`mnist_fm/models/lin_final.pth`](mnist_fm/models/lin_final.pth) — trained checkpoint (30 epochs, whole-module torch.save)
- [`mnist_fm_eval/results.json`](mnist_fm_eval/results.json) — all eval metrics
- [`mnist_fm_eval/samples_one_step.png`](mnist_fm_eval/samples_one_step.png) / [`samples_multi_step.png`](mnist_fm_eval/samples_multi_step.png) — 24-sample grids (visually indistinguishable)
- [`mnist_fm/artifacts/`](mnist_fm/artifacts/) — training-time sample grids every 5 epochs
- [`mnist_fm/history.json`](mnist_fm/history.json) — per-epoch losses (0.42 → 0.126)
- [`mnist_fm/code/Linearizer-official/`](mnist_fm/code/Linearizer-official/) — official repo snapshot @ `adfca2c`
- [`mnist_fm/args.json`](mnist_fm/args.json) — exact run configuration
## Rerun
```bash
uv run run_repro.py --epochs 30 --batch-size 128 --fid-samples 10000 \
--trackio-space <space_id> --out <out_dir>
```
Training dashboard: https://huggingface.co/spaces/abidlabs/linearizer-mnist-fm-trackio
## Provenance
Trained on Hugging Face Jobs (a10g-small) with trackio logging; driver imports the official
implementation verbatim from codeload tarball of commit `adfca2c`. arxiv:2510.08570

Xet Storage Details

Size:
5.03 kB
·
Xet hash:
f8acce8b6d84b25057c56a3f99371d8da05b34f346d6dd3a2780d2b26e4d9d0e

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.