| --- |
| library_name: mlx |
| license: mit |
| license_link: https://github.com/swz30/Restormer/blob/main/LICENSE.md |
| base_model: swz30/Restormer |
| pipeline_tag: image-to-image |
| tags: |
| - mlx |
| - image-restoration |
| - deblurring |
| - denoising |
| - restormer |
| --- |
| |
| # mlx-community/Restormer-defocus-deblurring-fp32 |
|
|
| [Restormer](https://github.com/swz30/Restormer) — **Single-image defocus deblur** — converted to **Apple MLX** for |
| Apple-Silicon inference via the |
| [`mlx-restormer-swift`](https://github.com/xocialize/mlx-restormer-swift) Swift package. |
|
|
| Zamir et al., *Restormer: Efficient Transformer for High-Resolution Image Restoration*, **CVPR |
| 2022**. **26,126,644 parameters** (494 tensors, `LayerNorm_type=WithBias`). |
|
|
| Out-of-focus deblur from a SINGLE image. DPDD 25.98. Notable because the high end of this task is licence-poisoned — IFAN, KPAC, DRBNet and LaKDNet are all AGPL-3.0 — so this is 0.44 dB off the best and the only permissively-licensed option. |
|
|
| > **Licence note.** Restormer is **plain MIT** with full commercial rights. The *same author's* |
| > MIRNet / MIRNetv2 / MPRNet / CycleISP carry an **Academic Public License** — easy to conflate, |
| > so it is worth stating explicitly. |
|
|
| ## Use with mlx-restormer-swift |
|
|
| ```swift |
| import RestormerMLXCore |
| |
| var cfg = Restormer.Configuration() |
| cfg.normKind = .withBias |
| let model = Restormer(cfg) |
| try model.loadWeights(from: weightsURL) |
| let restored = model.restoreTiled(imageNHWC) // NHWC RGB in [0,1] |
| ``` |
|
|
| Or as an MLXEngine `imageRestore` ModelPackage (`MLXRestormer.RestormerRestorePackage`), which |
| resolves this repo via the Hub. |
|
|
| ## ⚠️ Tile — do not run this full-frame |
|
|
| Measured on an M5 Max: a single 1080p frame full-frame costs **15.50 GB of MLX allocation / 48.02 GB |
| process footprint**. Tiled, the MLX peak is **flat at ~2.6 GB** from 512² to 1080p, because the peak |
| is one-tile-sized — a bigger image runs *more* tiles, not bigger ones. |
|
|
| **Tile geometry should be 8-aligned.** Three `pixelUnshuffle(2)` stages mean the 2×2 grouping grid is |
| measured from the tile origin at strides 2, 4 and 8 full-resolution pixels, so an unaligned origin |
| shifts that grid out of phase with the full-frame decomposition. |
|
|
| On overlap, measured by **seam visibility** rather than PSNR: overlap 0 leaves a faint but real seam |
| (boundary gradient 1.31× the interior), while every aligned overlap ≥ 8 measures clean (1.09–1.20×). |
| Note PSNR-against-full-frame actually *prefers* overlap 0 — which is why it is the wrong metric for |
| a tiler. |
|
|
| ## Conversion |
|
|
| MLX **NHWC** layout: conv `(O,I,kH,kW) → (O,kH,kW,I)` (186 standard + 88 depthwise), everything else |
| passthrough. The upstream `.pth` nests its state dict under `raw['params']` (BasicSR convention). |
|
|
| ## Parity |
|
|
| Gated against the PyTorch oracle on the CPU stream, fp32, judged on relative error: |
|
|
| - **key contract** — 494 tensors / 26,126,644 params / 0 missing / 0 unused, strict load |
| - **primitives** — both LayerNorm variants, and the pixel-shuffle pair (worst 2.6e-06) |
| - **blocks** — MDTA, GDFN, TransformerBlock (worst 2.6e-07) |
| - **full model** — cosine **1.00000000** at 64² / 128² / 256² (worst 1.0e-06) |
|
|
| The pixel-shuffle gate also runs the *wrong* channel ordering — the `(r,r,C)` split that compiles |
| and silently scrambles the image — and measures it at **493,451× the observed rounding**, so the |
| tolerance keeps ~5 orders of discrimination. |
|
|
| ## Honest limitation |
|
|
| Restormer's attention is spatially **global**, so it is weak on *spatially varying* blur. Benchmark |
| rank is also a weak predictor of real-world value: *"Deblurring in the Wild"* (2026) found every |
| GoPro-trained method scored below the blurry input on real smartphone blur. Validate on your own |
| photographs. |
|
|
| Weights: MIT (swz30/Restormer, first-party GitHub release v1.0). Port code: MIT. |
|
|
