README.md

---
license: other
pipeline_tag: image-to-image
tags:
- android
- neural-graphics
- gaming
- graphics
language:
- en
---

# Neural Super Sampling (NSS)

Neural Super Sampling (NSS) is an innovative, efficient network for temporal super sampling on mobile devices. Content rendered at 540p can be upscaled to 1080p, resulting in up to 50% GPU savings. With our retraining tools content creators and game studios can build derivatives of the model suited to artwork style and performance requirements.

### 🎥 Neural Super Sampling Demo

<video controls width="100%">
  <source src="https://huggingface.co/Arm/neural-super-sampling/resolve/main/resources/Enchanted_Castle_NSS_Demo.mp4" type="video/mp4">
  Your browser does not support the video tag.
</video>

## Model Details

Neural Super Sampling (NSS) is a parameter prediction model for real-time temporal super sampling developed by Arm, optimized for execution on Neural Accelerators (NX) in mobile GPUs. It enables high-resolution rendering at a lower compute cost by reconstructing high-quality output frames from low-resolution temporal inputs. NSS is particularly suited for mobile gaming, XR, and other power-constrained graphics use cases.

- **Developed by:** Arm Limited
- **Model type:** Temporal image super sampling
- **License:** Other
- **Repository:** [Neural Graphics Model Gym](https://github.com/arm/neural-graphics-model-gym)
- **Paper:** [How Neural Super Sampling Works](https://community.arm.com/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/how-arm-neural-super-sampling-works)
- **Quickstart with ML extensions for Vulkan®**: [ML extensions for Vulkan® Quickstart Guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/vulkan-ml-sample/)
- **Quickstart with Unreal**: [Neural Super Sampling Quickstart Guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/nss-unreal/) for NSS integration into Unreal Engine

NSS is under active development with regular updates planned. Follow Arm to get the latest.

## Uses

NSS can be directly integrated into graphics pipelines using ML extensions for Vulkan®. See included ML SDK for Vulkan [scenario](https://huggingface.co/Arm/neural-super-sampling/tree/main/scenario) for the simplest way to evaluate the model. The scenario includes the necessary pre- and post-processing compute shaders along with a single frame worth of input data.

The recommended way of integrating the model into a graphics pipeline is by using the [VGF Library](https://github.com/arm/ai-ml-sdk-vgf-library/tree/main) from the ML SDK for Vulkan. 

NSS is released under a [permissive license](https://huggingface.co/Arm/neural-super-sampling/blob/main/Arm_AI_Model_Community_License_v1_0_PRE-1154.pdf) designed to foster innovation in the graphics industry and provide differentiation to content creators.

### Direct Use

NSS has been integrated into Unreal Engine via two plugins, the [NSS Plugin for Unreal Engine](https://github.com/arm/nss-unreal-plugin/) and [Unreal NNE Plugin for ML extensions for Vulkan](https://github.com/arm/ml-extensions-for-vulkan-unreal-plugin/). See our [quick start guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/nss-unreal/) for step-by-step instructions on how to use NSS in Unreal® Engine.

### Out-of-Scope Use

- Not suited for non-temporal tasks such as a standalone image upsampling

## Bias, Risks, and Limitations

- Requires accurate motion vectors and frame history for stable output
- May underperform in extremely low framerate scenarios (<10 FPS) with fast camera movement
- Padding of the input is needed if input dimensions are not divisible by 8

### Recommendations

For ultra-low-FPS use cases, reduce the camera speed, acceleration, or both so that the relative motion between frames mimics the
application running at a higher frame rate.

## How to Get Started with the Model

This repository contains pre-trained weights and compiled NSS model in VGF format ready for integration with Vulkan applications.

The included Scenario demonstrates full execution of the model on a Vulkan compute-capable system. An Emulation Layer is provided to implement ML Extensions for Vulkan where it is not supported by the native Vulkan driver.

### Download and Prepare the Scenario

Clone the NSS model repository from Hugging Face
```powershell
git clone https://huggingface.co/Arm/TestNSS
cd TestNSS
```

### Run the Scenario

The NSS Hugging Face repository includes pre-built Windows® binaries for ML Emulation Layer for Vulkan and Scenario Runner. For other platforms,
- build from source following the instructions for [Building the Emulation Layer from source](https://github.com/arm/ai-ml-emulation-layer-for-vulkan/blob/main/README.md#building-the-emulation-layer-from-source) and [Building the Scenario Runner from source](https://github.com/arm/ai-ml-sdk-scenario-runner/blob/main/README.md#building-scenario-runner-from-source)
- adapt the instructions below accordingly 

1. Set the required environment variables:

   On Windows:
   ```powershell
   $env:VK_LAYER_PATH="$PWD\bin\windows-x86_64"
   $env:VK_INSTANCE_LAYERS="VK_LAYER_ML_Graph_Emulation;VK_LAYER_ML_Tensor_Emulation"
   ```

   On Linux (assuming the Emulation Layer binaries and JSON files and Scenario Runner executable are copied to `bin/linux-x86_64`):
   ```powershell
   export LD_LIBRARY_PATH=$PWD/bin/linux-x86_64:$LD_LIBRARY_PATH
   export VK_LAYER_PATH=$PWD/bin/linux-x86_64
   export VK_INSTANCE_LAYERS=VK_LAYER_ML_Graph_Emulation:VK_LAYER_ML_Tensor_Emulation
   ```
   
2. Execute the scenario:

   On Windows:
   ```powershell
   bin\windows-x86_64\scenario-runner.exe --scenario scenario\scenario.json --output out
   ```

   On Linux:
   ```powershell
   bin/linux-x86_64/scenario-runner --scenario scenario/scenario.json --output out
   ```
   
- Output images are encoded as `B10G11R11_UFLOAT`. This format is common for framebuffers but not widely supported by image viewers. Use [RenderDoc](https://renderdoc.org/) to view these images.

## Training and Evaluation

For background on NSS architecture and training read our blog: [How Neural Super Sampling Works](https://community.arm.com/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/how-arm-neural-super-sampling-works)

Training and evaluation details, including model architecture code, training pipeline, and test configurations, are available at:

- Model training code: https://github.com/arm/neural-graphics-model-gym
- Examples and tutorials: https://github.com/arm/neural-graphics-model-gym-examples
- Sample dataset: https://huggingface.co/datasets/Arm/neural-graphics-dataset

### 🔎 Model Explorer VGF extension

The [VGF extension to Model Explorer](https://github.com/arm/vgf-adapter-model-explorer) provides a simple interface to visualize model and analyse VGF composition. 

![Model Explorer screenshot](resources/model-explorer-screenshot.png)

## License

- The license for the model source code can be found [here](https://github.com/arm/neural-graphics-model-gym/blob/main/LICENSES/Apache-2.0.txt).
- The license for the content of this repository can be found [here](https://huggingface.co/Arm/neural-super-sampling/blob/main/Arm_AI_Model_Community_License_v1_0_PRE-1154.pdf)

## More Information

🧑‍🔬 More technical details about the model can be found in the [NSS Guide](https://developer.arm.com/documentation/110290/latest/).

👩🏽‍💻 Our [Neural Graphics Development Kit](https://developer.arm.com/mobile-graphics-and-gaming/neural-graphics) contains engine plugins, model training tools, code examples and extensive developer documentation.

🙋🏻‍♀️ For questions or feedback please [start a discussion](https://huggingface.co/Arm/neural-super-sampling/discussions)

## Trademark notice
Arm® is a registered trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere.

Windows® is a trademark of the Microsoft group of companies.

Vulkan® is a registered trademark of the [Khronos® Group](https://www.khronos.org/legal/trademarks).