| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: image-to-image |
| tags: |
| - video-super-resolution |
| - super-resolution |
| - apple-silicon |
| - videotoolbox |
| - research |
| --- |
| |
| # DetailNet r2 |
|
|
| A small recurrent network (about 1M parameters) that adds fine texture on top of the output of **Apple's on-device video super-resolution**, VideoToolbox's `VTSuperResolutionScaler`. It's for upscaling 720p video to 4K (×3) on Apple silicon. It was trained entirely on a MacBook Pro (M5 Max, 64 GB). |
|
|
| > **Research release. By eye, r2 is not a visible improvement over Apple AI.** On our metrics it is as steady as Apple's output, handles scene cuts more cleanly, and is closer to ground truth on LPIPS. But the extra fine detail it adds (1.13× Apple's) is too small to see, and it missed its own goal of 1.5×. It's published so others can build on the approach, not as a better upscaler to use today. |
|
|
| - **Code, benchmark and training pipeline:** https://github.com/tsonubin/metafx-upscale |
| - **License:** Apache-2.0 |
|
|
| ## How it works |
|
|
| For each frame, at source resolution (for example 1280×720), DetailNet takes: |
| - `x_t`, `x_prev`: the current and previous source frames (RGB, 0–1); |
| - `base`: Apple AI's output for the frame, resized to exactly 3× the source (RGB, 0–1); |
| - `flow4`: VideoToolbox optical flow between `x_prev` and `x_t`, in source pixels, with channels `back_x, back_y, fwd_x, fwd_y`; |
| - its own previous texture and hidden state, warped along the flow so the invented texture stays locked to the scene. |
|
|
| It returns `detail`, a zero-mean, high-pass texture layer at 3× resolution. The final frame is `base + strength · detail`, so the model can't change brightness, color or shapes. Its state is reset at scene cuts and wherever the flow is invalid. |
|
|
| | | | |
| |---|---| |
| | Architecture | conv3×3 → 12 residual blocks (64 channels) → texture head (pixel shuffle ×3, high-pass) + hidden-state head (32 channels, tanh) | |
| | Parameters | 972,795 (3.9 MB, float32) | |
| | Scale | ×3 only (720p → 4K). It needs Apple AI's base, so it runs on macOS 26+ | |
| | Cost | About 0.03–0.04 s per frame on MPS for a 704×424 source crop, on top of Apple AI's ≈0.35 s per frame | |
|
|
| ## Results |
|
|
| These are go / no-go gates on the project's benchmark (12 clips from broadcast nature documentaries), compared with `apple-base`, Apple AI's output before compression. |
| - **Detail** and **jitter** are ratios to Apple AI, measured on five real 720p → 4K clips. Jitter is motion-compensated frame-to-frame change; lower is steadier. |
| - **LPIPS** is measured on five ground-truth clips: real 1080p frames shrunk to 640×360, upscaled ×3 and compared with the original. Lower is better. |
|
|
| | Gate | Target | r2 | Apple AI | |
| |---|---|---|---| |
| | Fine detail | ≥ 1.5× | **1.13×** (fail) | 1.00× | |
| | Jitter, all motion | ≤ 1.1× | 1.00× | 1.00× | |
| | Jitter static / slow / fast | ≤ 1.1× each | 0.94 / 0.97 / 1.07 | 1.00 | |
| | Pulse (4-frame shimmer) | ≤ 1.1 | 1.01 | n/a | |
| | Detail leaking across a cut | ≤ Apple's | **0.14** | 0.52 | |
| | LPIPS vs real 1080p (×3) | ≤ Apple's | **0.252** | 0.317 | |
|
|
| Its measured wins come mostly from smoothing away Apple's slightly painterly artifacts, which LPIPS rewards. Side by side at 1:1, the two are hard to tell apart. Scaling the texture up 3× reaches 1.59× detail, but jitter rises to 1.28×, so the texture isn't locked well enough to amplify. |
|
|
| ## Usage |
|
|
| ```python |
| from safetensors.torch import load_file |
| from detailnet.model import DetailNet, compose # from the GitHub repo |
| |
| net = DetailNet(scale=3, channels=64, blocks=12, hidden=32).eval() |
| net.load_state_dict(load_file("model.safetensors")) |
| |
| prev_detail, hidden = net.zero_state(1, h, w, "cpu") # reset at the first frame and at every cut |
| detail, hidden = net(x_t, x_prev, base, flow4, prev_detail, hidden) |
| frame = compose(base, detail, strength=1.0) # base + strength * detail, clamped to [0, 1] |
| prev_detail = detail |
| ``` |
|
|
| **Producing the inputs:** |
| - The repo's Swift CLI writes `x_t`, `base` and `flow4` as frame dumps: `metalfx-upscale input.mp4 out.mp4 --engine ml --size 3x --dump-source S --dump-base B --dump-flow F`. |
| - The exact reset and flow-cleaning rules are in `detailnet/run_bench.py` and `detailnet/train.py` (`run_sequence`). |
| - There is no end-to-end "upscale my video with DetailNet" tool yet. The repo runs it over benchmark clips; putting it into the upscaler and app was planned as the next milestone. |
|
|
| ## Training |
|
|
| - **Data:** about 1,100 short sequences (720p and 1080p) cut from broadcast nature documentaries (ABC, Australian Broadcasting Corporation). The footage is copyrighted and isn't distributed; only the trained weights are released. |
| - **Teacher:** FlashVSR v1.1, a one-step streaming diffusion super-resolution model (Apache-2.0). We ported it to Apple's MPS and ran it offline to make texture targets. |
| - **Losses:** |
| - perceptual (LPIPS / DISTS) plus L1, against the teacher; |
| - a temporal lock term that penalizes texture changes not explained by optical flow; |
| - a ping-pong consistency term. |
| - **Schedule:** |
| - 14,000 iterations, batch 4, 160×90 source crops, learning rate 2e-4 with a cosine schedule, seed 1; |
| - initialised from r1's checkpoint at iteration 18,000; |
| - about 10 hours per round on the M5 Max. |
|
|
| ## Limitations |
|
|
| - **Not a visible improvement over Apple AI** (see above). Treat it as a baseline for further work, for example an adversarial loss or a stronger temporal lock. |
| - **Hard dependencies:** ×3 only, and it needs Apple AI's output and VideoToolbox optical flow, so macOS 26+ on Apple silicon. |
| - **Narrow training data:** one footage domain only (nature documentaries) and SDR only. |
|
|
| ## Attribution |
|
|
| - **Teacher:** FlashVSR v1.1 (Apache-2.0). |
| - **Base upscaler:** Apple VideoToolbox `VTSuperResolutionScaler`, which runs on the user's Mac; nothing from Apple is redistributed. |
| - **Footage:** used for training only, not distributed. |
|
|
