Upload README.md with huggingface_hub

d85cf7b verified 4 days ago

3.79 kB

library_name: mlx
license: mit
license_link: https://github.com/swz30/Restormer/blob/main/LICENSE.md
base_model: swz30/Restormer
pipeline_tag: image-to-image
tags:
  - mlx
  - image-restoration
  - deblurring
  - denoising
  - restormer

mlx-community/Restormer-defocus-deblurring-fp32

Restormer — Single-image defocus deblur — converted to Apple MLX for Apple-Silicon inference via the mlx-restormer-swift Swift package.

Zamir et al., Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 2022. 26,126,644 parameters (494 tensors, LayerNorm_type=WithBias).

Out-of-focus deblur from a SINGLE image. DPDD 25.98. Notable because the high end of this task is licence-poisoned — IFAN, KPAC, DRBNet and LaKDNet are all AGPL-3.0 — so this is 0.44 dB off the best and the only permissively-licensed option.

Licence note. Restormer is plain MIT with full commercial rights. The same author's MIRNet / MIRNetv2 / MPRNet / CycleISP carry an Academic Public License — easy to conflate, so it is worth stating explicitly.

Use with mlx-restormer-swift

import RestormerMLXCore

var cfg = Restormer.Configuration()
cfg.normKind = .withBias
let model = Restormer(cfg)
try model.loadWeights(from: weightsURL)
let restored = model.restoreTiled(imageNHWC)   // NHWC RGB in [0,1]

Or as an MLXEngine imageRestore ModelPackage (MLXRestormer.RestormerRestorePackage), which resolves this repo via the Hub.

⚠️ Tile — do not run this full-frame

Measured on an M5 Max: a single 1080p frame full-frame costs 15.50 GB of MLX allocation / 48.02 GB process footprint. Tiled, the MLX peak is flat at ~2.6 GB from 512² to 1080p, because the peak is one-tile-sized — a bigger image runs more tiles, not bigger ones.

Tile geometry should be 8-aligned. Three pixelUnshuffle(2) stages mean the 2×2 grouping grid is measured from the tile origin at strides 2, 4 and 8 full-resolution pixels, so an unaligned origin shifts that grid out of phase with the full-frame decomposition.

On overlap, measured by seam visibility rather than PSNR: overlap 0 leaves a faint but real seam (boundary gradient 1.31× the interior), while every aligned overlap ≥ 8 measures clean (1.09–1.20×). Note PSNR-against-full-frame actually prefers overlap 0 — which is why it is the wrong metric for a tiler.

Conversion

MLX NHWC layout: conv (O,I,kH,kW) → (O,kH,kW,I) (186 standard + 88 depthwise), everything else passthrough. The upstream .pth nests its state dict under raw['params'] (BasicSR convention).

Parity

Gated against the PyTorch oracle on the CPU stream, fp32, judged on relative error:

key contract — 494 tensors / 26,126,644 params / 0 missing / 0 unused, strict load
primitives — both LayerNorm variants, and the pixel-shuffle pair (worst 2.6e-06)
blocks — MDTA, GDFN, TransformerBlock (worst 2.6e-07)
full model — cosine 1.00000000 at 64² / 128² / 256² (worst 1.0e-06)

The pixel-shuffle gate also runs the wrong channel ordering — the (r,r,C) split that compiles and silently scrambles the image — and measures it at 493,451× the observed rounding, so the tolerance keeps ~5 orders of discrimination.

Honest limitation

Restormer's attention is spatially global, so it is weak on spatially varying blur. Benchmark rank is also a weak predictor of real-world value: "Deblurring in the Wild" (2026) found every GoPro-trained method scored below the blurry input on real smartphone blur. Validate on your own photographs.

Weights: MIT (swz30/Restormer, first-party GitHub release v1.0). Port code: MIT.