README.md

metadata

license: mit
library_name: onnxruntime
pipeline_tag: image-to-image
tags:
  - super-resolution
  - upscaling
  - onnx
  - real-time
  - nrr

NRR upscalers - the released 2x model, and a 2.5x tier

A 2x neural upscaler as a single ONNX graph of about 461 KB, trained in the NRR repository and used there as the released model: the one the Unreal, Godot and Unity plugins load, the one docs/parity.md measures, and the one this repository's headless verifications render.

The same repository publishes a 2.5x model under ratio25/, which is what the 768x432 -> 1920x1080 tier needs. The released 2x model cannot be asked for that tier: the ratio is the graph's own head, so fed a 768x432 frame it returns 1536x864, and it reaches 1920x1080 only from a 960x540 input.

It is a prototype model from a research repository rather than a shipping upscaler. It is published so the plugin side of that repository can be verified against a real model instead of a sample.

The graph

inputs	color - the low-resolution frame; jitter - the sub-pixel offset the frame was rendered at (all zeros for a render that does not jitter); scale - the tier token
output	one frame, 2x the input on both axes
spatial	dynamic H/W: the graph takes whatever size the input is
the scale token	0.0 for the 128 -> 256 grid, 1.0 for the 256 -> 512 grid. The model was trained at both, so it is measured at both - docs/parity.md and docs/parity-512.md are two halves of one result rather than two results. NRR's own harness derives the token from the frame's width, so each tier is handed the value it was trained with
depth, motion	not inputs. This model declares neither; those belong to the temporal models in the repository, not to this one

Measured quality (this repository's own harness, not a vendor's)

From docs/parity.md - all val scenes (models/training-data/godot-v6-warp, 48 frames, 2.00x), on an NVIDIA GeForce RTX 4070 Ti, driver 610.88. Every arm below was driven from the scene's own frames and scored against that scene's own targets:

arm	tier	PSNR dB	SSIM	MS-SSIM	LPIPS	DISTS	VMAF	detail
bilinear	128x128->256x256	26.3563	0.87165	0.95257	0.1025	0.2732	36.793	0.2634
bicubic	128x128->256x256	25.7689	0.86524	0.94596	0.0936	0.2795	40.399	0.4258
lanczos	128x128->256x256	25.4109	0.85991	0.94447	0.1025	0.2846	40.912	0.4459
fsr1 (EASU + RCAS)	128x128->256x256	24.6755	0.84641	0.93548	0.0947	0.2958	42.255	0.7668
nrr - this model	128x128->256x256	26.4320	0.88151	0.95515	0.0570	0.1609	43.374	0.5616
DLSS (NGX)	128x128->256x256	26.6829	0.88217	0.96077	0.1047	0.2576	52.253	0.2378
XeSS 2.0.2	128x128->256x256	26.7147	0.87228	0.95586	0.1319	0.2718	31.847	0.1774

That is a comparison of these frames at this tier, which is what it is. No row here is a general claim about any upscaler, and a timing taken on a host doing other work is a timing on a busy host. The repository's docs/evaluation-protocol.md states the rules the table is held to.

The 2.5x tier (ratio25/)

ratio25/upscale_25x_procedural_depth.onnx upscales by 2.5 on both axes, and it is the first model in that repository trained at a ratio that is not a power of two and qualified at it by the repository's own gates - a model that fails one is refused, not exported with a warning next to it.

inputs	color - the low-resolution frame; depth - per-pixel ray distance on the input's grid
output	one frame, 2.5x the input on both axes (768x432 -> 1920x1080)
spatial	dynamic H/W: the graph takes whatever size the input is
parameters	90,974 (369,025 bytes)
sha256	86326c5d7c4ca5b7b5e7b5e206b55de2909f80a7fee7d1938d3d661545c545d9
how the ratio is reached	PixelShuffle(5) synthesises 25 polyphase components per colour channel, a learned stride-2 convolution selects the phase, and the skip is the 2.5x bilinear upsample - so nothing outside the graph resamples the frame, and an untrained model is that ratio's bilinear baseline

Measured on the run's own held-out split (120 procedural pairs, color being the noisy packed input the runtime supplies), with the 2.5x bilinear baseline scored on the same frames by the same code:

arm	L1	SSIM	PSNR dB	MS-SSIM
bilinear at 2.5x	0.02542	0.8897	21.55	0.9427
this model	0.01058	0.9479	27.38	0.9739

58.37% below the baseline on the metric the training gates score (the bar is 5%), with training progress 58.9% and a depth ablation of 0.02742 - the gate that refuses an input the model does not actually use. Through the runtime at the tier: 224.6 ms/frame on an RTX 4070 Ti (CUDA provider, 102.1 ms of it inference) and 2104.3 ms on the same host's CPU execution provider; the benchmark seeds a synthetic depth ramp for depth-consuming models, so those rows measure speed and the quality numbers above come from the trainer's held-out split instead.

What is not claimed about the procedural model. Its data is generated (tools/gen_training_pairs.py --ratio 2.5: each scene rendered twice, at 128 and at 320, so the target is a real render at the ratio rather than a resample onto it), so its 58.37% is an in-family number: held-out pairs, but the same scenes, the same renderer and no held-out content. It is not comparable to the 2x table below either, which is measured on captured frames by a different harness at a different tier.

The same ratio trained on captured frames (ratio25/)

These are the out-of-family numbers: the same ratio, trained on frames captured from Godot - the content pipeline the released 2x model was trained on - over 840 pairs (360 train / 480 held out) of seven scenes the model has never seen.

model	inputs	parameters	held-out L1 vs the 2.5x bilinear baseline	SSIM	PSNR dB	MS-SSIM
ratio25/upscale_25x_captured_nj_depth_s2.onnx	color, depth	90,974	0.01359 vs 0.01516 → 10.37% better	0.9035 vs 0.8947	26.81 vs 27.22	0.9715 vs 0.9624
ratio25/upscale_25x_captured_depthjitter.onnx	color, depth, jitter	93,430	0.01399 vs 0.01541 → 9.24% better	0.9008 vs 0.8936	27.66 vs 27.10	0.9689 vs 0.9600

Three things a reader should know before choosing between them:

• The color,depth model loses PSNR. It is 0.41 dB below the bilinear baseline (26.81 against 27.22) while gaining on L1, SSIM and MS-SSIM. The training gates score L1, and the column that disagrees is shown rather than dropped. The color,depth,jitter model is above the baseline on all four metrics.
• The color,depth configuration is seed-fragile on captured content: 1 of 4 seeds qualified. One seed froze at the bilinear plane and then diverged (residual L1 0.01756 → 0.29974), one reached 1.9% progress with its output 0.000302 from the plane, one reached 2.5%. The published weights are the seed that qualified (20261112). The spread is part of the result - the procedural configuration in the section above qualified on its first seed with 58.37%.
• The phase is what makes a jittered capture learnable. On the jittered capture - the low-resolution sample moving up to half a pixel between frames - color,depth measured val L1 equal to the baseline's to five decimals (0.01541 against 0.01541), with an output 0.000155 from the plane: a half-pixel misalignment the model was never told about has the bilinear plane as its L1 optimum. Declaring jitter on the same frames reached 9.24%, with both declared inputs measurably used (depth 0.00540, jitter 0.00695).

Both graphs are trained at 128 -> 320 and are dynamic in H/W, so both render 768x432 -> 1920x1080; the runtime rows are docs/measurements/ratio25-captured-* in the repository, and the training configuration, the captures and the packing script are there too.

One thing to know before comparing this model's output across hardware: ONNX Runtime's CUDA execution provider computes convolutions in TF32 by default, so this graph's CPU and CUDA outputs differ by up to 6.341e-04 (mean 1.558e-05) while agreeing to 2.176e-06 with use_tf32=0. The two recorded runtime rows therefore carry different frame hashes for a named reason rather than a mysterious one, and a re-run on one provider reproduces its hash exactly.

Files

file	bytes	what it is
upscale_msreal_scale.onnx	461,501	the released model: the two-tier (scale token) upscaler described above
ratio25/upscale_25x_procedural_depth.onnx	369,025	the 2.5x model: 768x432 -> 1920x1080, inputs color, depth
ratio25/upscale_25x_procedural_depth.report.json	2,934	that model's gate report: the configuration, the sha256, the held-out numbers and the baselines they were scored against - the repository gitignores models/**/*.json, so this copy is the one that travels with the weights
ratio25/upscale_25x_captured_nj_depth_s2.onnx	369,025	the same ratio trained on captured frames: inputs color, depth, 10.37% below its ratio's bilinear baseline on 480 held-out captured pairs
ratio25/upscale_25x_captured_nj_depth_s2.report.json	2,955	its gate report, including the baselines every number was scored against
ratio25/upscale_25x_captured_depthjitter.onnx	384,090	the same capture with the phase declared: inputs color, depth, jitter, 9.24% better and above the baseline on all four metrics
ratio25/upscale_25x_captured_depthjitter.report.json	2,991	its gate report
phases/upscale_subsampled.onnx	424,038	the 128 -> 256 model docs/evaluation-protocol.md names when it explains why a model must be judged at the tier it was trained for
phases/upscale_mid.onnx	424,038	mid-training model, kept because the phase tables compare against it
phases/upscale_mid_regression.onnx	424,038	the mid model as a regression baseline
probes/*.onnx	790 - 45,082	the sample model and the small probe graphs the test suite and the plugin drift guards load (nrr_upscaler_v0.1, nrr_passthrough_2x, nrr_scale_token_probe, nrr_unrecognised_input_probe, nrr_history_probe)
tests/p4_tjit_20261025.onnx	499,854	the reference model the Unity verification project renders with: its tests score their output against engine_plugins/unity_verify/Assets/StreamingAssets/smoke_fixture/fixture.json, whose recorded reference means were produced by this exact file. The tests select the highest-dated models/p5/p4_tjit_*.onnx, so this is a real training checkpoint - one of the history, whose other 118 checkpoints are in SamurAI-Official/NRR-checkpoints

Using it

With NRR itself:

nrr_model_load(device, "upscale_msreal_scale.onnx", &model);
nrr_render(device, model, /*references=*/NULL, &input, &output);   /* the runtime derives scale from the width */

The third argument is the reference set, not the inputs: NRR hands a model the planes it declares, and takes them from the frame - NRRFrameInput.color / .depth / .motion_vectors, NRRTemporalState.jitter and .previous_output for a temporal model, and scale derived from the frame's width. That is the whole difference between the two contracts on this page. The released 2x model declares color, jitter, scale, which the runtime already has; the 2.5x model declares color and depth, so a caller has to populate input.depth:

nrr_model_load(device, "ratio25/upscale_25x_procedural_depth.onnx", &model);
nrr_render(device, model, /*references=*/NULL, &input, &output);   /* input.depth carries the frame's depth */

Fetching it into a checkout, which puts it exactly where the repository expects it (engine_plugins/unreal_verify/setup.ps1 and the test suite load it from there):

hf download SamurAI-Official/NRR upscale_msreal_scale.onnx --local-dir models/phase4

Anywhere else it is a plain ONNX graph:

import onnxruntime
session = onnxruntime.InferenceSession("upscale_msreal_scale.onnx")

Provenance, and what is not claimed

Trained in the NRR repository's own trainer on frames captured from Godot; the training configuration, the comparison harness, the plugins that load it and every number above live in that repository. The released model is not tracked by git there (models/phase4/ is working state), which is why it is here.

The 2.5x model in ratio25/ was trained in the same repository, by tools/run_ratio25.ps1 on data it generates rather than data it captured (tools/gen_training_pairs.py --ratio 2.5), and its gate report travels with it because the repository gitignores models/**/*.json - without that copy the artifact's provenance would live only in a changelog. The weights are tracked there, so the two copies can be compared byte for byte.

Verification status, stated because a model that loads is not a model that ran: NRR's Unreal project renders this model at three tiers headless and prints RESULT: PASS (engine_plugins/unreal_verify/README.md), and the Godot addon is verified end to end (CHANGELOG.md). Those runs are on a host where the CUDA execution provider crashes during model load - a host/driver result, recorded in that README - so the numbers there are CPU numbers.

One description in the repository does not match this model: models/architecture.md there still describes an aspirational 1080p -> 4K color+depth+motion network, which is not what is published here. The contract above is this graph's real one, read from the model.

Licence: MIT - see LICENSE.