restore model card (accidental overwrite) + benchmark update

README.md

@@ -1,20 +1,387 @@
 ---
-title: NERVE
-colorFrom: indigo
-colorTo: green
-sdk: gradio
-sdk_version: 5.25.2
-app_file: app.py
-pinned: false
-suggested_hardware: zero-a10g
+license: apache-2.0
+language:
+- en
+library_name: pytorch
+pipeline_tag: image-to-image
+tags:
+- super-resolution
+- image-restoration
+- image-to-image
+- sisr
+- real-world-restoration
+- pytorch
+- onnx
+- ncnn
+- spandrel
+- chainner
+- edge
+- mobile
+datasets:
+- Phips/lucid-cc0-v2-hc-512
 ---
 
-# NERVE — image upscaling & restoration
+# NERVE
 
-A tiny demo for the released [NERVE](https://huggingface.co/Phips/NERVE) models.
+**Norm-free Efficient Restoration for Various Edge devices**
 
-- **2x / 4x release** — for clean images.
-- **4x OTF GAN** — for degraded images (blurry, noisy, JPEG-compressed).
+A lightweight super-resolution architecture that is simple to train and fast to
+run. Released models are **verified** with **traiNNer-redux** (training) and
+**ONNX Runtime** + **ncnn** (inference). The op set is standard, so other
+runtimes (**CoreML**, **TensorRT**, DirectML, ...) are expected to work too —
+but they are not tested here.
 
-Input: an image. Output: an uncompressed PNG you can download.
-Runs on ZeroGPU; large images are processed in tiles.
+Part of the **BODY** suite by **Philip Hofmann** — networks built to be
+real-world usable first, not benchmark-chasing. See
+[Relation to HEART](#relation-to-heart) for which one to pick.
+
+> Short version: copy `nerve_arch.py` into traiNNer-redux, train with the sample
+> configs, then run the `.safetensors`, `.onnx`, or `.ncnn` models. One
+> architecture, one size, no fused/unfused pairs. Details below.
+
+![NERVE 4x on anime — input | bicubic | release | OTF GAN](examples/ani_compare.png)
+*Input (as-is) | bicubic x4 | NERVE 4x release | NERVE 4x OTF GAN.*
+
+![NERVE 4x on a noisy image](examples/noisy_compare.png)
+![NERVE 4x on a Stable-Diffusion image](examples/stablediffusion_compare.png)
+
+---
+
+## Why NERVE?
+
+There are strong and established SR networks already. This is the honest case
+for when NERVE is worth your time — and when it is not.
+
+This project comes out of a trainer's frustration rather than a benchmark goal.
+After training hundreds of SISR models (see
+[Phhofm/models](https://github.com/Phhofm/models)), the recurring annoyance was
+networks that felt like *paper* networks: they chase leaderboard metrics with
+techniques that leave you juggling an unfused and a fused checkpoint, or whose
+dynamic-ONNX conversion is fragile. This network is the opposite of that — built
+so a trainer can pick it up, train it, and convert it to dynamic ONNX without
+headaches.
+
+**Strengths of NERVE**
+- **Real-world usability over metric chasing.** Trained on a CC0 dataset with
+  the full Real-ESRGAN OTF degradation pipeline; the OTF GAN model restores
+  degraded images, not just clean bicubic downsamples.
+- **One file, one variant.** A single readable architecture file and one model
+  size. No S/M/L matrix, no fused-vs-unfused checkpoints, nothing to fuse before
+  deploy — what you train is what you ship.
+- **Export just works.** Dynamic-shape ONNX (verified, ORT parity ~1e-5) and a
+  clean ncnn conversion. Standard ops only (`Conv / Add / ReLU / DepthToSpace /
+  Resize`).
+- **Boring, stable training.** Norm-free, bf16 AMP, no gradient clipping, a
+  single Charbonnier loss for the base stage.
+- **Small and fast.** ~1.8M params — SPAN-class size but conv-only, suited to
+  mobile, edge, web and game engines.
+- **Maintainable for years.** One op set, no exotic dependencies, ~85-line arch.
+
+**When to pick something else.** If the goal is maximum PSNR/LPIPS on academic
+benchmarks, or a large transformer, NERVE is not aiming there — use HEART, HAT,
+ATD, DAT2, or one of the community networks below. NERVE aims at simplicity,
+deployability and hassle-free training.
+
+**Credit where it is due.** None of this exists in a vacuum. SISR has many
+strong architectures and generous community members, for example HAT, ATD and
+DAT2, the community network work of [umzi](https://github.com/umzi2) and
+[Kim2091](https://github.com/Kim2091), fdat by
+[sharekhan](https://github.com/stinkybread), and
+[the-database](https://github.com/the-database) (traiNNer-redux, its benchmarks
+and its models). This list is far from complete — apologies to anyone not named;
+the SISR community has a great many people contributing a great deal.
+
+---
+
+## Quickstart
+
+### As a user (just want to upscale / restore images)
+
+Pick a model and run it:
+
+| I want to... | Use |
+|---|---|
+| upscale a clean image 4x | `models/4x_NERVE_release.safetensors` |
+| upscale a clean image 2x | `models/2x_NERVE_release.safetensors` |
+| restore a blurry / noisy / JPEG-y image 4x | `models/4x_NERVE_OTF_gan.safetensors` |
+| run on-device / inside an engine | the `onnx/` or `ncnn/` files |
+
+- **ONNX Runtime** (verified): use `onnx/*.onnx` (dynamic H/W). TensorRT /
+  DirectML should work too (same op set) but are untested here.
+- **ncnn** (verified, fp16): `ncnn/*.ncnn.param` + `*.ncnn.bin`. Feed images with
+  `Mat.from_pixels`:
+  ```python
+  import ncnn
+  net = ncnn.Net(); net.load_param("4x_NERVE_fp16.ncnn.param"); net.load_model("4x_NERVE_fp16.ncnn.bin")
+  ex = net.create_extractor()
+  ex.input("in0", ncnn.Mat.from_pixels(bgr_uint8, ncnn.Mat.PixelType.PIXEL_BGR2RGB, w, h))
+  out = ex.extract("out0")[1]   # CHW float, 4x
+  ```
+  (Use `from_pixels`; the ncnn Python binding's raw-array constructor misbehaves
+  on large inputs. Outputs are fp16, so expect small deviations at hard edges.)
+- **chaiNNer**: load the `onnx/*.onnx` files directly. Loading the
+  `.safetensors` via spandrel needs NERVE registered in your chaiNNer/spandrel
+  build (not upstream yet).
+- Input: RGB, any size. Output: 2x or 4x.
+
+### As a trainer (want to train your own)
+
+NERVE is a normal [traiNNer-redux](https://github.com/the-database/traiNNer-redux)
+architecture — no special steps.
+
+```bash
+# 1. Add the architecture + its ICNR helper (auto-registers: traiNNer scans *_arch.py)
+cp nerve_arch.py <traiNNer-redux>/traiNNer/archs/
+cp icnr.py       <traiNNer-redux>/traiNNer/utils/   # ICNR init helper (not in upstream yet)
+
+# 2. Copy a training config and point it at your data
+cp configs/4x_NERVE_release.yml <traiNNer-redux>/options/
+#    -> edit dataroot_gt / dataroot_lq in that file
+
+# 3. Train
+cd <traiNNer-redux>
+python train.py -opt options/4x_NERVE_release.yml --auto_resume
+
+# 4. Export to optimized dynamic ONNX (official converter)
+cp configs/4x_NERVE_onnx.yml <traiNNer-redux>/options/
+#    -> set path.pretrain_network_g to your trained checkpoint
+python convert_to_onnx.py -opt options/4x_NERVE_onnx.yml
+```
+
+That is the whole workflow — 2 small files plus a config. The net is one file,
+the config is one file, the checkpoint is one file — **no fusion step, no paired
+fused/unfused weights.** (`icnr.py` is the only piece not in upstream
+traiNNer-redux yet; it is the ICNR initializer for the PixelShuffle head.)
+
+- `4x_NERVE_onnx.yml` exports dynamic-shape ONNX (opset 20) and can emit
+  `fp32`, `fp16`, or `bf16` (`onnx.dtype`), with OnnxSlim optimization and a
+  built-in PyTorch-vs-ONNX verify pass.
+- Prefer no extra dependencies? `scripts/export_dynamic.py` is a tiny
+  standalone exporter (torch + onnxruntime only).
+
+**Start from a released pretrain** (skip the expensive early training) by
+setting `path.pretrain_network_g` in your training config to a file from
+`models/`:
+
+| training stage | warm-start from |
+|---|---|
+| `4x_NERVE_release.yml` | `models/4x_NERVE_release.safetensors` (or the 2x one with `strict_load_g: false`) |
+| `4x_NERVE_OTF_fidelity.yml` | `models/4x_NERVE_release.safetensors` |
+| `4x_NERVE_OTF_gan.yml` | your OTF fidelity checkpoint |
+
+**Training tips** (all NERVE models follow these):
+- bf16 AMP is fine; no gradient clipping needed (norm-free net).
+- `--auto_resume` is recommended for long runs.
+- For real-world input, follow the chain: clean release -> OTF fidelity -> OTF GAN.
+
+---
+
+## Models
+
+| File | Role | Notes |
+|---|---|---|
+| `models/2x_NERVE_release.safetensors` | **2x official pretrain** | clean bicubic, 31.84 dB / 0.9255 SSIM (Urban100) |
+| `models/4x_NERVE_release.safetensors` | **4x official pretrain** | clean bicubic, 25.09 dB / 0.7531 SSIM (Urban100) |
+| `models/4x_NERVE_OTF_fidelity.safetensors` | 4x OTF fidelity | restore degraded input; the pretrain for OTF GAN finetunes |
+| `models/4x_NERVE_OTF_gan.safetensors` | 4x OTF GAN | **sharpest real-world output** (topiq 0.430 / lpips 0.290, Urban100 OTF val) |
+| `onnx/4x_NERVE_1x3xHxW_fp32_op20.onnx` | 4x release, dynamic ONNX | onnxslim-optimized, fp32 |
+| `onnx/2x_NERVE_1x3xHxW_fp32_op20.onnx` | 2x release, dynamic ONNX | onnxslim-optimized, fp32 |
+| `onnx/4x_NERVE_OTF_fidelity_1x3xHxW_fp32_op20.onnx` | 4x OTF fidelity, dynamic ONNX | onnxslim-optimized, fp32 |
+| `onnx/4x_NERVE_OTF_gan_1x3xHxW_fp32_op20.onnx` | 4x OTF GAN, dynamic ONNX | onnxslim-optimized, fp32 |
+| `ncnn/4x_NERVE_fp16.ncnn.param` + `.bin` | 4x release, ncnn | fp16 weights |
+| `ncnn/2x_NERVE_fp16.ncnn.param` + `.bin` | 2x release, ncnn | fp16 weights |
+| `ncnn/4x_NERVE_OTF_fidelity_fp16.ncnn.param` + `.bin` | 4x OTF fidelity, ncnn | fp16 weights |
+| `ncnn/4x_NERVE_OTF_gan_fp16.ncnn.param` + `.bin` | 4x OTF GAN, ncnn | fp16 weights |
+
+`models/*.safetensors` are plain EMA state dicts (no wrapper) — load them with
+`nerve_arch.nerve(scale=N)`.
+
+All released models (2x/4x release, 4x OTF fidelity, 4x OTF GAN) are final.
+
+**Pretrain chain** (how the models relate):
+
+```
+2x_NERVE_release ──→ 4x_NERVE_release ──→ OTF fidelity ──→ OTF GAN
+```
+
+The release models are the shared starting point for anyone training NERVE:
+warm-start from them (2x → 4x, or release → OTF fidelity → OTF GAN) to skip the
+expensive early training.
+
+---
+
+## Examples
+
+Real images are fed in **as-is** (not downscaled) and upscaled 4x. For each
+image there are two comparisons:
+
+- `examples/*_compare.png` — whole image at **full resolution**: **input** |
+  **bicubic x4** | **NERVE 4x release** | **NERVE 4x OTF GAN**.
+- `examples/*_1to1.png` — a **1:1 native-pixel center crop**.
+
+The bicubic column is the baseline; the OTF GAN column is the released showcase
+model. The source inputs are in `examples/inputs/`.
+
+- `examples/realphoto_compare.png` (street scene)
+- `examples/ani_compare.png` (anime)
+- `examples/manga_compare.png` (manga)
+- `examples/text_compare.png` (text)
+- `examples/noisy_compare.png` (noisy input)
+- `examples/stablediffusion_compare.png` (AI-generated)
+
+## Architecture
+
+```
+5x5 conv stem → 24 × (Conv-ReLU-Conv + residual) → PixelShuffle → + bicubic(x)
+```
+
+- **~1.8M params**, pure convolution — **no attention, no normalization, no
+  gating, no positional encoding, no reparameterization.**
+- A **bicubic input residual** means the net only learns what bicubic upscaling
+  misses — that's why it stays small and trains fast.
+- **ICNR-initialized** PixelShuffle head (checkerboard-free by construction).
+- The op graph is trivial: `Conv, Add, ReLU, DepthToSpace, Resize`. That's the
+  entire reason it exports to dynamic ONNX and converts to ncnn (verified)
+  without custom ops or fused/unfused pairs; CoreML is expected but untested.
+
+## What we actually tested
+
+Every design choice here came from a measured ablation, not a guess. The full
+development log — including dead ends and raw numbers — is in
+[`docs/ABLATIONS.md`](docs/ABLATIONS.md). Highlights:
+
+- **Normalization**: removing LayerNorm beat the normed baseline by **+1.15 dB**
+  at ~2x the training speed -> norm-free.
+- **Depth/width sweep + capacity scaling**: quality improved with size; the
+  sweep put the GAN quality knee at **1.80M** (0.69M -> 1.80M: lpips -0.015;
+  1.80M -> 2.39M: only -0.002 more).
+- **Dilation**: a dilated-conv A/B under the OTF recipe was **null** -> plain convs.
+- **Checkerboard**: diagnosed a PixelShuffle phase-lock, fixed with ICNR plus
+  *keeping aliased (nearest-exact) LR*. An anti-aliased-LR + sub-pixel-jitter
+  variant made thin lines *worse* and was reverted.
+- **OTF/GAN capability**: the small net trained productively under the full
+  Real-ESRGAN OTF GAN recipe (no collapse).
+- **Runtime validation**: every released `.safetensors` loads (strict) and runs;
+  every `.onnx` loads in ONNX Runtime; the ncnn models were run at multiple input
+  sizes up to 720x720 (2880x2880 output).
+- **Deployability**: dynamic-ONNX export + ORT parity and ncnn conversion were
+  verified at every stage (the numbers are in the log).
+- **Not tested here** (community welcome): TensorRT / DirectML *speed* and
+  fixed-shape static exports, CoreML, and on-device benchmarks. Train more
+  variants, benchmark, and share — the arch is standard ops only.
+
+## Training data & license
+
+All released NERVE models were trained **only** on
+[`Phips/lucid-cc0-v2-hc-512`](https://huggingface.co/datasets/Phips/lucid-cc0-v2-hc-512),
+a **CC0** dataset — the released weights are clean to use commercially.
+
+We did **not** optimize for benchmark leaderboards. If you want higher
+PSNR/LPIPS numbers, training on larger academic datasets (DIV2K, Flickr2K,
+LSDIR, ...) is expected to score better; NERVE is a normal architecture, train
+it on whatever you like.
+
+## Relation to HEART
+
+NERVE and HEART are the two members of the **BODY** suite (by Philip Hofmann):
+same philosophy (real-world usability > benchmark scores), different tiers.
+
+| | **HEART** | **NERVE** |
+|---|---|---|
+| size | ~16.7M params | ~1.8M params |
+| design | attention (FlashAttention-friendly) | pure convolution |
+| use when | you want the **best quality** and have the compute (desktop, server, high-end device) | you want **speed, small size, and dead-simple deployment** (mobile, edge, web, game engines) |
+| cost | heavier to run | a fraction of the cost |
+
+Rule of thumb: **HEART for maximum quality, NERVE for maximum practicality.**
+Both export to dynamic ONNX; NERVE additionally converts to ncnn cleanly
+because it has no attention ops.
+
+Side-by-side comparisons on the same inputs (`input | bicubic | both release
+models | both OTF GAN models`), each with a 1:1 crop:
+`examples/OST_009_nerve_vs_heart_*` and `examples/00003_nerve_vs_heart_*`
+(Real-ESRGAN test set), plus `examples/realphoto_nerve_vs_heart_*` and
+`examples/noisy_nerve_vs_heart_*`.
+
+---
+
+## Repository layout
+
+```
+nerve_arch.py                    the architecture (copy to traiNNer/archs/)
+icnr.py                          ICNR init helper (copy to traiNNer/utils/; not in upstream yet)
+configs/                         example training configs (+ 4x_NERVE_onnx.yml for ONNX export)
+models/                          pretrained checkpoints (see table above)
+onnx/                            dynamic fp32 ONNX (opset 20, onnxslim-optimized)
+ncnn/                            ncnn fp16 param/bin
+docs/ABLATIONS.md                the full development/ablation log (what we tested)
+examples/                        real-image comparisons (input vs bicubic vs release vs GAN)
+scripts/export_dynamic.py        checkpoint → dynamic ONNX exporter (+ ORT verify)
+LICENSE                          Apache-2.0
+```
+
+## Help wanted (community welcome)
+
+The goal here is **real-world usability** — easy training, hassle-free dynamic
+ONNX, simple and robust to maintain — **not** leaderboard chasing. So NERVE's
+strength is *simplicity for trainers*; there's still plenty worth measuring.
+Pick anything below and open an issue/PR/discussion with your numbers.
+
+**Benchmarks** (deliberately not chased here)
+- Train on an academic dataset (DIV2K / Flickr2K / LSDIR) and compare against
+  SPAN, SRVGGNetCompact, HAT, Real-ESRGAN (PSNR / SSIM / LPIPS).
+- Perceptual metrics (LPIPS / TopIQ / MUSIQ) on real-world degraded images.
+
+**Speed & deployment** (not benchmarked here)
+- TensorRT / DirectML latency + peak VRAM on the dynamic ONNX.
+- Static-shape ONNX (e.g. `1x3x256x256`) vs dynamic: speed + memory.
+- fp16 vs fp32 ONNX.
+- ncnn on-device (Android / iOS): latency + memory.
+- CoreML conversion + on-device test.
+- Mobile CPU (XNNPACK / ORT Mobile / ncnn).
+
+**Models & code**
+- More finetunes (illustration, manga, denoise, 1x restoration).
+- Architecture experiments — see [`docs/ABLATIONS.md`](docs/ABLATIONS.md) for
+  what we already tested (and rejected).
+
+Small, measured contributions are welcome.
+
+## License
+
+Apache-2.0 (this repository's code and pretrained weights).
+
+## Update — 2026-09-22: speed benchmark (honest numbers)
+
+We ran a head-to-head **inference benchmark on an RTX 3060** using the
+traiNNer-redux benchmark protocol (input `1x3x480x640`, fp16, 50 runs), against
+the closest networks by parameter count:
+
+| model | params | PyTorch fp16 | TensorRT fp16 | ncnn (Vulkan GPU) |
+|---|---|---|---|---|
+| **NERVE 2x** | 1.78M | 90.0 ms | 35.3 ms | 117 ms |
+| SPAN_S 2x | 2.22M | 40.4 ms | **16.8 ms** | — |
+| PLKSR_tiny 2x | 2.35M | 110.1 ms | 380.6 ms | — |
+| compact 2x | 0.60M | 35.7 ms | 19.1 ms | — |
+
+**NERVE does not win on speed.** SPAN_S is ~2.1x faster at a similar parameter
+count, and compact is faster still. (PLKSR_tiny is slower in PyTorch and falls
+apart in ONNX/TensorRT because its large-kernel/CCM ops don't map, so NERVE does
+beat it there.)
+
+That is neither a surprise nor a problem for what NERVE is for. **NERVE is not a
+benchmark-chasing network.** It exists for ease of use and real-world usability:
+one variant, one file, no fused/unfused checkpoint pairs, and a checkpoint that
+converts to dynamic-shape ONNX, TensorRT and ncnn with no special handling.
+Speed is good enough, not the goal.
+
+Practical notes from the benchmark:
+- **TensorRT is the fastest runtime for NERVE** (~2.5x over PyTorch), and it
+  wants a **static shape** — that is where static exports actually matter.
+- In ONNX Runtime, static vs dynamic made no measurable difference.
+- fp16 ≈ bf16.
+- **ncnn runs on the GPU via Vulkan** (117 ms vs 5.4 s on CPU), but that is the
+  Python binding; native ncnn on a phone is the intended path, not a desktop race.
+
+Choose SPAN if you want maximum speed. Choose NERVE if you want the simplest
+thing to train, convert and ship.