mlx-community/Restormer-real-denoising-fp32

Restormer — Real denoise — converted to Apple MLX for Apple-Silicon inference via the mlx-restormer-swift Swift package.

Zamir et al., Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 2022. 26,111,668 parameters (406 tensors, LayerNorm_type=BiasFree).

Real-world denoise (SIDD / DND). Carries the only solid DND 40.03, i.e. it generalizes across sensors rather than to SIDD's five smartphone cameras.

Licence note. Restormer is plain MIT with full commercial rights. The same author's MIRNet / MIRNetv2 / MPRNet / CycleISP carry an Academic Public License — easy to conflate, so it is worth stating explicitly.

Use with mlx-restormer-swift

import RestormerMLXCore

var cfg = Restormer.Configuration()
cfg.normKind = .biasFree
let model = Restormer(cfg)
try model.loadWeights(from: weightsURL)
let restored = model.restoreTiled(imageNHWC)   // NHWC RGB in [0,1]

Or as an MLXEngine imageRestore ModelPackage (MLXRestormer.RestormerRestorePackage), which resolves this repo via the Hub.

⚠️ This checkpoint is BiasFree — it has a different key set

Restormer's denoising models normalize LayerNorm_type='BiasFree', which has no bias vectors at all: 406 tensors / 26,111,668 params, against the deblur checkpoints' 494 / 26,126,644. Build the model with the wrong norm type and the strict load fails with exactly 88 missing keys.

BiasFree also does not centre its output — it is x / sqrt(var + eps) · weight, with the variance computed about the mean but no mean subtraction. Implementing it as "LayerNorm minus the bias term" is wrong.

⚠️ Tile — do not run this full-frame

Measured on an M5 Max: a single 1080p frame full-frame costs 15.50 GB of MLX allocation / 48.02 GB process footprint. Tiled, the MLX peak is flat at ~2.6 GB from 512² to 1080p, because the peak is one-tile-sized — a bigger image runs more tiles, not bigger ones.

Tile geometry should be 8-aligned. Three pixelUnshuffle(2) stages mean the 2×2 grouping grid is measured from the tile origin at strides 2, 4 and 8 full-resolution pixels, so an unaligned origin shifts that grid out of phase with the full-frame decomposition.

On overlap, measured by seam visibility rather than PSNR: overlap 0 leaves a faint but real seam (boundary gradient 1.31× the interior), while every aligned overlap ≥ 8 measures clean (1.09–1.20×). Note PSNR-against-full-frame actually prefers overlap 0 — which is why it is the wrong metric for a tiler.

Conversion

MLX NHWC layout: conv (O,I,kH,kW) → (O,kH,kW,I) (186 standard + 88 depthwise), everything else passthrough. The upstream .pth nests its state dict under raw['params'] (BasicSR convention).

Parity

Gated against the PyTorch oracle on the CPU stream, fp32, judged on relative error:

• key contract — 406 tensors / 26,111,668 params / 0 missing / 0 unused, strict load
• primitives — both LayerNorm variants, and the pixel-shuffle pair (worst 2.6e-06)
• blocks — MDTA, GDFN, TransformerBlock (worst 2.6e-07)
• full model — cosine 1.00000000 at 64² / 128² / 256² (worst 1.0e-06)

The pixel-shuffle gate also runs the wrong channel ordering — the (r,r,C) split that compiles and silently scrambles the image — and measures it at 493,451× the observed rounding, so the tolerance keeps ~5 orders of discrimination.

Honest limitation

Restormer's attention is spatially global, so it is weak on spatially varying blur. Benchmark rank is also a weak predictor of real-world value: "Deblurring in the Wild" (2026) found every GoPro-trained method scored below the blurry input on real smartphone blur. Validate on your own photographs.

Weights: MIT (swz30/Restormer, first-party GitHub release v1.0). Port code: MIT.