---
license: apache-2.0
library_name: pytorch
pipeline_tag: image-to-image
tags:
  - video-super-resolution
  - super-resolution
  - apple-silicon
  - videotoolbox
  - research
---

# DetailNet r2

A small recurrent network (about 1M parameters) that adds fine texture on top of the output of **Apple's on-device video super-resolution**, VideoToolbox's `VTSuperResolutionScaler`. It's for upscaling 720p video to 4K (×3) on Apple silicon. It was trained entirely on a MacBook Pro (M5 Max, 64 GB).

> **Research release. By eye, r2 is not a visible improvement over Apple AI.** On our metrics it is as steady as Apple's output, handles scene cuts more cleanly, and is closer to ground truth on LPIPS. But the extra fine detail it adds (1.13× Apple's) is too small to see, and it missed its own goal of 1.5×. It's published so others can build on the approach, not as a better upscaler to use today.

- **Code, benchmark and training pipeline:** https://github.com/tsonubin/metafx-upscale
- **License:** Apache-2.0

## How it works

For each frame, at source resolution (for example 1280×720), DetailNet takes:
- `x_t`, `x_prev`: the current and previous source frames (RGB, 0–1);
- `base`: Apple AI's output for the frame, resized to exactly 3× the source (RGB, 0–1);
- `flow4`: VideoToolbox optical flow between `x_prev` and `x_t`, in source pixels, with channels `back_x, back_y, fwd_x, fwd_y`;
- its own previous texture and hidden state, warped along the flow so the invented texture stays locked to the scene.

It returns `detail`, a zero-mean, high-pass texture layer at 3× resolution. The final frame is `base + strength · detail`, so the model can't change brightness, color or shapes. Its state is reset at scene cuts and wherever the flow is invalid.

| | |
|---|---|
| Architecture | conv3×3 → 12 residual blocks (64 channels) → texture head (pixel shuffle ×3, high-pass) + hidden-state head (32 channels, tanh) |
| Parameters | 972,795 (3.9 MB, float32) |
| Scale | ×3 only (720p → 4K). It needs Apple AI's base, so it runs on macOS 26+ |
| Cost | About 0.03–0.04 s per frame on MPS for a 704×424 source crop, on top of Apple AI's ≈0.35 s per frame |

## Results

These are go / no-go gates on the project's benchmark (12 clips from broadcast nature documentaries), compared with `apple-base`, Apple AI's output before compression.
- **Detail** and **jitter** are ratios to Apple AI, measured on five real 720p → 4K clips. Jitter is motion-compensated frame-to-frame change; lower is steadier.
- **LPIPS** is measured on five ground-truth clips: real 1080p frames shrunk to 640×360, upscaled ×3 and compared with the original. Lower is better.

| Gate | Target | r2 | Apple AI |
|---|---|---|---|
| Fine detail | ≥ 1.5× | **1.13×** (fail) | 1.00× |
| Jitter, all motion | ≤ 1.1× | 1.00× | 1.00× |
| Jitter static / slow / fast | ≤ 1.1× each | 0.94 / 0.97 / 1.07 | 1.00 |
| Pulse (4-frame shimmer) | ≤ 1.1 | 1.01 | n/a |
| Detail leaking across a cut | ≤ Apple's | **0.14** | 0.52 |
| LPIPS vs real 1080p (×3) | ≤ Apple's | **0.252** | 0.317 |

Its measured wins come mostly from smoothing away Apple's slightly painterly artifacts, which LPIPS rewards. Side by side at 1:1, the two are hard to tell apart. Scaling the texture up 3× reaches 1.59× detail, but jitter rises to 1.28×, so the texture isn't locked well enough to amplify.

## Usage

```python
from safetensors.torch import load_file
from detailnet.model import DetailNet, compose   # from the GitHub repo

net = DetailNet(scale=3, channels=64, blocks=12, hidden=32).eval()
net.load_state_dict(load_file("model.safetensors"))

prev_detail, hidden = net.zero_state(1, h, w, "cpu")        # reset at the first frame and at every cut
detail, hidden = net(x_t, x_prev, base, flow4, prev_detail, hidden)
frame = compose(base, detail, strength=1.0)                  # base + strength * detail, clamped to [0, 1]
prev_detail = detail
```

**Producing the inputs:**
- The repo's Swift CLI writes `x_t`, `base` and `flow4` as frame dumps: `metalfx-upscale input.mp4 out.mp4 --engine ml --size 3x --dump-source S --dump-base B --dump-flow F`.
- The exact reset and flow-cleaning rules are in `detailnet/run_bench.py` and `detailnet/train.py` (`run_sequence`).
- There is no end-to-end "upscale my video with DetailNet" tool yet. The repo runs it over benchmark clips; putting it into the upscaler and app was planned as the next milestone.

## Training

- **Data:** about 1,100 short sequences (720p and 1080p) cut from broadcast nature documentaries (ABC, Australian Broadcasting Corporation). The footage is copyrighted and isn't distributed; only the trained weights are released.
- **Teacher:** FlashVSR v1.1, a one-step streaming diffusion super-resolution model (Apache-2.0). We ported it to Apple's MPS and ran it offline to make texture targets.
- **Losses:**
  - perceptual (LPIPS / DISTS) plus L1, against the teacher;
  - a temporal lock term that penalizes texture changes not explained by optical flow;
  - a ping-pong consistency term.
- **Schedule:**
  - 14,000 iterations, batch 4, 160×90 source crops, learning rate 2e-4 with a cosine schedule, seed 1;
  - initialised from r1's checkpoint at iteration 18,000;
  - about 10 hours per round on the M5 Max.

## Limitations

- **Not a visible improvement over Apple AI** (see above). Treat it as a baseline for further work, for example an adversarial loss or a stronger temporal lock.
- **Hard dependencies:** ×3 only, and it needs Apple AI's output and VideoToolbox optical flow, so macOS 26+ on Apple silicon.
- **Narrow training data:** one footage domain only (nature documentaries) and SDR only.

## Attribution

- **Teacher:** FlashVSR v1.1 (Apache-2.0).
- **Base upscaler:** Apple VideoToolbox `VTSuperResolutionScaler`, which runs on the user's Mac; nothing from Apple is redistributed.
- **Footage:** used for training only, not distributed.