DetailNet r2: weights, config and model card

README.md

@@ -0,0 +1,99 @@
+---
+license: apache-2.0
+library_name: pytorch
+pipeline_tag: image-to-image
+tags:
+  - video-super-resolution
+  - super-resolution
+  - apple-silicon
+  - videotoolbox
+  - research
+---
+
+# DetailNet r2
+
+A small recurrent network (about 1M parameters) that adds fine texture on top of the output of **Apple's on-device video super-resolution**, VideoToolbox's `VTSuperResolutionScaler`. It's for upscaling 720p video to 4K (×3) on Apple silicon. It was trained entirely on a MacBook Pro (M5 Max, 64 GB).
+
+> **Research release. By eye, r2 is not a visible improvement over Apple AI.** On our metrics it is as steady as Apple's output, handles scene cuts more cleanly, and is closer to ground truth on LPIPS. But the extra fine detail it adds (1.13× Apple's) is too small to see, and it missed its own goal of 1.5×. It's published so others can build on the approach, not as a better upscaler to use today.
+
+- **Code, benchmark and training pipeline:** https://github.com/tsonubin/metafx-upscale
+- **License:** Apache-2.0
+
+## How it works
+
+For each frame, at source resolution (for example 1280×720), DetailNet takes:
+- `x_t`, `x_prev`: the current and previous source frames (RGB, 0–1);
+- `base`: Apple AI's output for the frame, resized to exactly 3× the source (RGB, 0–1);
+- `flow4`: VideoToolbox optical flow between `x_prev` and `x_t`, in source pixels, with channels `back_x, back_y, fwd_x, fwd_y`;
+- its own previous texture and hidden state, warped along the flow so the invented texture stays locked to the scene.
+
+It returns `detail`, a zero-mean, high-pass texture layer at 3× resolution. The final frame is `base + strength · detail`, so the model can't change brightness, color or shapes. Its state is reset at scene cuts and wherever the flow is invalid.
+
+| | |
+|---|---|
+| Architecture | conv3×3 → 12 residual blocks (64 channels) → texture head (pixel shuffle ×3, high-pass) + hidden-state head (32 channels, tanh) |
+| Parameters | 972,795 (3.9 MB, float32) |
+| Scale | ×3 only (720p → 4K). It needs Apple AI's base, so it runs on macOS 26+ |
+| Cost | About 0.03–0.04 s per frame on MPS for a 704×424 source crop, on top of Apple AI's ≈0.35 s per frame |
+
+## Results
+
+These are go / no-go gates on the project's benchmark (12 clips from broadcast nature documentaries), compared with `apple-base`, Apple AI's output before compression.
+- **Detail** and **jitter** are ratios to Apple AI, measured on five real 720p → 4K clips. Jitter is motion-compensated frame-to-frame change; lower is steadier.
+- **LPIPS** is measured on five ground-truth clips: real 1080p frames shrunk to 640×360, upscaled ×3 and compared with the original. Lower is better.
+
+| Gate | Target | r2 | Apple AI |
+|---|---|---|---|
+| Fine detail | ≥ 1.5× | **1.13×** (fail) | 1.00× |
+| Jitter, all motion | ≤ 1.1× | 1.00× | 1.00× |
+| Jitter static / slow / fast | ≤ 1.1× each | 0.94 / 0.97 / 1.07 | 1.00 |
+| Pulse (4-frame shimmer) | ≤ 1.1 | 1.01 | n/a |
+| Detail leaking across a cut | ≤ Apple's | **0.14** | 0.52 |
+| LPIPS vs real 1080p (×3) | ≤ Apple's | **0.252** | 0.317 |
+
+Its measured wins come mostly from smoothing away Apple's slightly painterly artifacts, which LPIPS rewards. Side by side at 1:1, the two are hard to tell apart. Scaling the texture up 3× reaches 1.59× detail, but jitter rises to 1.28×, so the texture isn't locked well enough to amplify.
+
+## Usage
+
+```python
+from safetensors.torch import load_file
+from detailnet.model import DetailNet, compose   # from the GitHub repo
+
+net = DetailNet(scale=3, channels=64, blocks=12, hidden=32).eval()
+net.load_state_dict(load_file("model.safetensors"))
+
+prev_detail, hidden = net.zero_state(1, h, w, "cpu")        # reset at the first frame and at every cut
+detail, hidden = net(x_t, x_prev, base, flow4, prev_detail, hidden)
+frame = compose(base, detail, strength=1.0)                  # base + strength * detail, clamped to [0, 1]
+prev_detail = detail
+```
+
+**Producing the inputs:**
+- The repo's Swift CLI writes `x_t`, `base` and `flow4` as frame dumps: `metalfx-upscale input.mp4 out.mp4 --engine ml --size 3x --dump-source S --dump-base B --dump-flow F`.
+- The exact reset and flow-cleaning rules are in `detailnet/run_bench.py` and `detailnet/train.py` (`run_sequence`).
+- There is no end-to-end "upscale my video with DetailNet" tool yet. The repo runs it over benchmark clips; putting it into the upscaler and app was planned as the next milestone.
+
+## Training
+
+- **Data:** about 1,100 short sequences (720p and 1080p) cut from broadcast nature documentaries (ABC, Australian Broadcasting Corporation). The footage is copyrighted and isn't distributed; only the trained weights are released.
+- **Teacher:** FlashVSR v1.1, a one-step streaming diffusion super-resolution model (Apache-2.0). We ported it to Apple's MPS and ran it offline to make texture targets.
+- **Losses:**
+  - perceptual (LPIPS / DISTS) plus L1, against the teacher;
+  - a temporal lock term that penalizes texture changes not explained by optical flow;
+  - a ping-pong consistency term.
+- **Schedule:**
+  - 14,000 iterations, batch 4, 160×90 source crops, learning rate 2e-4 with a cosine schedule, seed 1;
+  - initialised from r1's checkpoint at iteration 18,000;
+  - about 10 hours per round on the M5 Max.
+
+## Limitations
+
+- **Not a visible improvement over Apple AI** (see above). Treat it as a baseline for further work, for example an adversarial loss or a stronger temporal lock.
+- **Hard dependencies:** ×3 only, and it needs Apple AI's output and VideoToolbox optical flow, so macOS 26+ on Apple silicon.
+- **Narrow training data:** one footage domain only (nature documentaries) and SDR only.
+
+## Attribution
+
+- **Teacher:** FlashVSR v1.1 (Apache-2.0).
+- **Base upscaler:** Apple VideoToolbox `VTSuperResolutionScaler`, which runs on the user's Mac; nothing from Apple is redistributed.
+- **Footage:** used for training only, not distributed.

config.json

@@ -0,0 +1,30 @@
+{
+  "architecture": "DetailNet",
+  "release": "r2",
+  "scale": 3,
+  "channels": 64,
+  "blocks": 12,
+  "hidden": 32,
+  "parameters": 972795,
+  "training": {
+    "iters": 14000,
+    "batch": 4,
+    "lr": 0.0002,
+    "weight_decay": 0.0001,
+    "w_perc": 1.0,
+    "w_l1": 1.0,
+    "w_lock": 1.0,
+    "w_pp": 0.5,
+    "seed": 1,
+    "crop": [
+      160,
+      90
+    ],
+    "reset_prob": 0.15,
+    "pp_prob": 0.2,
+    "perc_frames": 4,
+    "grad_clip": 1.0,
+    "initialised_from": "r1 (iteration 18000)",
+    "device": "Apple M5 Max (MPS)"
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8188eff2ca81b3166f4814be8687dee65ccd5e570dbdb01484e97e4d00093784
+size 3895972