--- license: cc-by-nc-4.0 library_name: pytorch pipeline_tag: image-to-image tags: - denoising - image-restoration - photography - hdr - coreml - apple-silicon --- # Nagi Denoise — NagiV2-L A blind denoiser for **real high-ISO photographs**. Float32 linear-light RGB in, float32 linear-light RGB out. HDR-safe (values above 1.0 are preserved, never clipped), seam-free at any resolution including 40MP+, and fully deterministic — same bytes in, same bytes out. * **Architecture**: NagiV2-L, **15.43M parameters** (width 64, encoder (2,2,4), middle 6, decoder (1,2,2)). * **Code**: (Apache-2.0) * **Weights**: this repo (CC BY-NC 4.0 — see [Licence](#licence)) ## Files in this repo | file | size | what it is | |---|---|---| | `nagi_v2_l_ft2_final.pt` | 236 MB | the production PyTorch checkpoint | | `nagi_v2_l_ft2_t768_fp16.mlpackage/` | 30 MB | Core ML export, 768px tile, fp16 — the fast path | | `nagi_v2_l_ft2_t768_fp32.mlpackage/` | 59 MB | Core ML export, 768px tile, fp32 | The `.mlpackage` assets are directories, stored here as folders of files. ## Usage ```python import numpy as np from nagi_denoise import denoise # img: (H, W, 3) float32 numpy array, linear-light RGB. Values > 1.0 are fine. out = denoise(img) # PyTorch reference path out = denoise(img, backend="coreml") # Core ML fast path on Apple Silicon ``` The weights are resolved locally first and downloaded from this repo only if they are not already present: an explicit `weights=` path wins, then `$NAGI_DENOISE_WEIGHTS`, then the in-repo `runs/` copy, then an existing Hugging Face cache, and only then a download. `allow_download=False` (or `NAGI_DENOISE_OFFLINE=1`) forbids the network entirely. To fetch deliberately, ahead of time: ```python from nagi_denoise.assets import resolve_weights, resolve_coreml_package resolve_weights() # -> Path to nagi_v2_l_ft2_final.pt resolve_coreml_package() # -> Path to the fp16 .mlpackage ``` ## ⚠️ Core ML: never use `compute_units="all"` `ALL` lets Core ML dispatch to the Apple Neural Engine, and **the ANE computes this fp16 graph wrongly**. On every one of the 23 validation tiles, peak output values run **1.25x–4.8x higher** than the PyTorch reference — worst absolute per-pixel error **8.7 on a scene whose true peak is 5.3**. Visually this is blown-out garbage, not a subtle numerical difference. The same graph on `cpu_and_gpu` agrees with PyTorch to **0.018** max, which is just fp16 rounding. `cpu_and_gpu` is the default in this project and it must stay that way. ## Training data | source | weight | licence | |---|---|---| | SIDD (Smartphone Image Denoising Dataset) | 0.25 | MIT | | PolyU Real-World Noisy Images Dataset | 0.35 | **non-commercial** | | synthetic Poisson-Gaussian degradations, generated by this project | 0.40 | this project's code (Apache-2.0) | The synthetic noise is **spatially correlated**, not white. Real demosaiced sensor noise is correlated; training on white noise alone teaches the model that "isolated pixel deviation = noise", so it preserves real correlated noise blobs as if they were fine structure. Measured lag-1 autocorrelation of the residual after a 3x3 median: | source | lag-1 autocorrelation | |---|---| | Fujifilm X-T5 (real) | +0.235 | | Pentax K-5 (real) | +0.139 | | PolyU (real) | +0.27 … +0.48 | | SIDD (real, but near-white) | +0.024 | | naive white synthetic noise | −0.146 | The synthetic noise field is therefore drawn as a Gaussian-blurred, unit-std normal field, with the blur sigma sampled uniformly per image over **[0.0, 1.0]** pixels (chroma field: [0.8, 2.0]). The sweep used to calibrate it — sigma 0.5 / 0.7 / 0.9 → lag-1 0.005 / 0.217 / 0.365 — puts sigma ≈ 0.7 on the real X-T5 figure and sigma ≈ 0.9 in the PolyU range. ## Measured results | benchmark | result | |---|---| | **SIDD Validation (sRGB PSNR)** | **39.030 dB** | | — NAFNet-w64 teacher, for reference | 40.21 dB, at 116M parameters (7.5x the size) | **Structure-vs-noise selectivity**, X-T5 Occi hair ROI — how much more high-frequency energy survives on structure than on flat noise: | pipeline | selectivity | retention on structure | |---|---|---| | **NagiV2 (this model)** | **+11.2 pt** | **90.5%** | | legacy in-house v12 pipeline | +9.8 pt | 82.1% | More detail retained *and* cleaner — not a trade. **HDR highlight retention** ≥ 0.99 on every scene with true HDR content. **Speed**, 39.8MP frame, end to end, on an M1 with 16GB: | path | time | |---|---| | Core ML fp16 / `cpu_and_gpu` | **~83 s** | | pure PyTorch / MPS | ~261 s | ## Known limitations Stated plainly, because they are real: * **Isolated small specular highlights on low-dynamic-range files get treated as impulse noise.** On Z7 fix / Z7 bird, top-1% luma retention is **0.62–0.92**. The conditional highlight guard does not help here by design: it only arms when the image genuinely contains above-SDR content, because arming it on low-range scenes blends noisy input back in over large areas (measured 3.7x more flat-region noise) and reads as uneven denoising. * **The confidence-gated detail head is inert on this checkpoint.** The gate is closed, so `detail_strength` is a no-op by design. It is wired up so it activates automatically if a future checkpoint opens the gate. * **It removes noise; it does not generate texture.** This is a restoration model, not a generative one. It will not match the synthesised detail of DxO PhotoLab DeepPrime XD, and it is not trying to. ## Licence **The weights in this repo are CC BY-NC 4.0 — non-commercial.** The source code at is Apache-2.0. Full text: The weights are not more restrictive by choice. They are a derived work of their training data, and one of the sources carries a non-commercial restriction that flows through to anything trained on it: > PolyU Real-World Noisy Images Dataset > Copyright (c) 2018, The Hong Kong Polytechnic University > "Any redistribution, use, or modification is done solely for > non-commercial purposes." PolyU supplied **35%** of the training mixture, and it was not an incidental ingredient: it was **the only source of real spatially-correlated camera noise**, which is exactly what made the model work on real photographs at all. The other sources are permissive — SIDD is MIT, and the synthetic degradations are generated by this project's own code. You may: use the weights for any non-commercial purpose; redistribute them, modified or not; fine-tune, distil, quantise or convert them and redistribute the result. You must: give attribution, state that the weights are CC BY-NC 4.0, and indicate whether you modified them. Derivative weights inherit these terms. ### If you need commercially usable weights **Retrain without PolyU.** The code to do so is Apache-2.0 and complete: drop `polyu` from `data.mixture` in a training config, redistribute its weight between `sidd` and `synthetic`, and train per the repo's README. Expect a quality cost — removing the only real correlated-noise source is a material change and the result needs re-validating against the project's gates. Weights you produce that way are yours, and are not covered by this licence. ### No warranty The weights are provided "as is", without warranty of any kind, express or implied. See the CC BY-NC 4.0 text for the full disclaimer. ## Citation The datasets and reference models this work depends on: ```bibtex @inproceedings{abdelhamed2018sidd, title = {A High-Quality Denoising Dataset for Smartphone Cameras}, author = {Abdelhamed, Abdelrahman and Lin, Stephen and Brown, Michael S.}, booktitle = {CVPR}, year = {2018} } @article{xu2018polyu, title = {Real-world Noisy Image Denoising: A New Benchmark}, author = {Xu, Jun and Li, Hui and Liang, Zhetong and Zhang, David and Zhang, Lei}, year = {2018}, note = {PolyU Real-World Noisy Images Dataset. Non-commercial use only.} } @article{chen2022nafnet, title = {Simple Baselines for Image Restoration}, author = {Chen, Liangyu and Chu, Xiaojie and Zhang, Xiangyu and Sun, Jian}, journal = {arXiv:2204.04676}, year = {2022} } @article{zhang2022scunet, title = {Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis}, author = {Zhang, Kai and Li, Yawei and Liang, Jingyun and Cao, Jiezhang and Zhang, Yulun and Tang, Hao and Timofte, Radu and Van Gool, Luc}, year = {2022} } ``` NAFNet (MIT) is vendored in the repo as the distillation teacher and benchmark reference; SCUNet (Apache-2.0) as a benchmark reference. Full third-party notices are in `NOTICE` in the source repo.