---
license: apache-2.0
pipeline_tag: image-to-image
library_name: pytorch
tags:
  - super-resolution
  - image-enhancement
  - generative-modeling
---

# Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution (GSASR)

This repository contains the official models and code for **GSASR**, a novel method for arbitrary-scale super-resolution, presented in the paper [Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution](https://huggingface.co/papers/2501.06838).

📚 [Paper](https://huggingface.co/papers/2501.06838) | 🌐 [Project Page](https://mt-cly.github.io/GSASR.github.io/) | 💻 [Code](https://github.com/ChrisDud0257/GSASR) | 🚀 [Gradio Demo](https://huggingface.co/spaces/mutou0308/GSASR)

## Abstract
Implicit Neural Representations (INR) have been successfully employed for Arbitrary-scale Super-Resolution (ASR). However, INR-based models need to query the multi-layer perceptron module numerous times and render a pixel in each query, resulting in insufficient representation capability and low computational efficiency. Recently, Gaussian Splatting (GS) has shown its advantages over INR in both visual quality and rendering speed in 3D tasks, which motivates us to explore whether GS can be employed for the ASR task. However, directly applying GS to ASR is exceptionally challenging because the original GS is an optimization-based method through overfitting each single scene, while in ASR we aim to learn a single model that can generalize to different images and scaling factors. We overcome these challenges by developing two novel techniques. Firstly, to generalize GS for ASR, we elaborately design an architecture to predict the corresponding image-conditioned Gaussians of the input low-resolution image in a feed-forward manner. Each Gaussian can fit the shape and direction of an area of complex textures, showing powerful representation capability. Secondly, we implement an efficient differentiable 2D GPU/CUDA-based scale-aware rasterization to render super-resolved images by sampling discrete RGB values from the predicted continuous Gaussians. Via end-to-end training, our optimized network, namely GSASR, can perform ASR for any image and unseen scaling factors. Extensive experiments validate the effectiveness of our proposed method.

## Overview

GSASR achieves state-of-the-art in arbitrary-scale super-resolution by representing given low-resolution images as millions of continuous 2D Gaussians.

![Fast Rasterization](https://github.com/ChrisDud0257/GSASR/raw/main/assets/sampling.png)

## Performance

**Comparisons with representative/SoTA ASR models (PSNR/SSIM are tested on Y channel of Ycbcr space).**

<div style="overflow-x:auto; font-size:10px;">
<table>
  <tr>
    <th rowspan="2">Encoder Backbone</th>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Version</th>
    <th rowspan="2">Training Dataset</th>
    <th colspan="3" align="center">PSNR/SSIM/LPIPS/DIST (x4 scaling factor)</th>
  </tr>
  <tr>
    <td align="center">DIV2K</td>
    <td align="center">LSDIR</td>
    <td align="center">Urban100</td>
  </tr>

  <!-- EDSR Backbone -->
  <tr>
    <td rowspan="6">EDSR</td>
    <td>LIIF</td>
    <td>Paper</td>
    <td>DIV2K</td>
    <td align="center">30.43/0.8388/0.2662/0.1403</td>
    <td align="center">26.21/0.7614/0.2978/0.1678</td>
    <td align="center">26.14/0.7885/0.2271/0.1738</td>
  </tr>
  <tr>
    <td>GaussianSR</td>
    <td>Paper</td>
    <td>DIV2K</td>
    <td align="center">30.46/0.8389/0.2684/0.1406</td>
    <td align="center">26.23/0.7615/0.3007/0.1679</td>
    <td align="center">26.19/0.7893/0.2283/0.1730</td>
  </tr>
  <tr>
    <td>CiaoSR</td>
    <td>Paper</td>
    <td>DIV2K</td>
    <td align="center">30.67/0.8431/0.2585/0.1370</td>
    <td align="center">26.42/0.7681/0.2865/0.1631</td>
    <td align="center">26.69/0.8091/0.2078/0.1659</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Paper Reported</td>
    <td>DIV2K</td>
    <td align="center">30.89/0.8486/0.2518/0.1301</td>
    <td align="center">26.65/0.7774/0.2777/0.1554</td>
    <td align="center">27.01/0.8142/0.1987/0.1552</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Enhanced</td>
    <td>DIV2K</td>
    <td align="center">31.01/0.8509/0.2508/0.1306</td>
    <td align="center">26.78/0.7813/0.2962/0.1543</td>
    <td align="center">27.34/0.8230/0.1920/0.1515</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Enhanced</td>
    <td>DF2K</td>
    <td align="center">31.04/0.8515/0.2512/0.1307</td>
    <td align="center">26.82/0.7827/0.2751/0.1540</td>
    <td align="center">27.45/0.8256/0.1902/0.1507</td>
  </tr>

  <!-- RDN Backbone -->
  <tr>
    <td rowspan="6">RDN</td>
    <td>LIIF</td>
    <td>Paper</td>
    <td>DIV2K</td>
    <td align="center">30.71/0.8449/0.2566/0.1354</td>
    <td align="center">26.48/0.7714/0.2838/0.1603</td>
    <td align="center">26.71/0.8055/0.2062/0.1562</td>
  </tr>
  <tr>
    <td>GaussianSR</td>
    <td>Paper</td>
    <td>DIV2K</td>
    <td align="center">30.76/0.8457/0.2570/0.1347</td>
    <td align="center">26.53/0.7727/0.2837/0.1595</td>
    <td align="center">26.77/0.8064/0.2069/0.1610</td>
  </tr>
  <tr>
    <td>CiaoSR</td>
    <td>Paper</td>
    <td>DIV2K</td>
    <td align="center">30.91/0.8481/0.2525/0.1327</td>
    <td align="center">26.66/0.7770/0.2768/0.1563</td>
    <td align="center">27.10/0.8142/0.1966/0.1559</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Paper Reported</td>
    <td>DIV2K</td>
    <td align="center">30.96/0.8500/0.2505/0.1288</td>
    <td align="center">26.73/0.7801/0.2752/0.1533</td>
    <td align="center">27.15/0.8177/0.1953/0.1515</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Enhanced</td>
    <td>DIV2K</td>
    <td align="center">31.03/0.8513/0.2499/0.1306</td>
    <td align="center">26.79/0.7819/0.2740/0.1543</td>
    <td align="center">27.37/0.8238/0.1898/0.1511</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Enhanced</td>
    <td>DF2K</td>
    <td align="center">31.10/0.8525/0.2482/0.1296</td>
    <td align="center">26.88/0.7848/0.2709/0.1527</td>
    <td align="center">27.58/0.8289/0.1849/0.1500</td>
  </tr>

  <!-- SWIN Backbone -->
  <tr>
    <td rowspan="4">SWIN</td>
    <td>CiaoSR</td>
    <td>Paper</td>
    <td>DIV2K</td>
    <td align="center">31.05/0.8511/0.2487/0.1316</td>
    <td align="center">26.80/0.7812/0.2724/0.1552</td>
    <td align="center">27.40/0.8231/0.1869/0.1535</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Paper (not Reported)</td>
    <td>DIV2K</td>
    <td align="center">31.06/0.8521/0.2487/0.1270</td>
    <td align="center">26.84/0.7837/0.2719/0.1503</td>
    <td align="center">27.39/0.8247/0.1913/0.1466</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Enhanced</td>
    <td>DIV2K</td>
    <td align="center">31.10/0.8530/0.2463/0.1285</td>
    <td align="center">26.88/0.7849/0.2690/0.1517</td>
    <td align="center">27.55/0.8280/0.1850/0.1475</td>
  </tr>
  <tr>
    <td>GSASR</td>
    <td>Enhanced</td>
    <td>DF2K</td>
    <td align="center">31.17/0.8541/0.2456/0.1288</td>
    <td align="center">26.96/0.7876/0.2665/0.1513</td>
    <td align="center">27.81/0.8343/0.1781/0.1465</td>
  </tr>

  <!-- HATL Backbone -->
  <tr>
    <td>HATL</td>
    <td>GSASR</td>
    <td>Ultra Performance</td>
    <td>SA1B</td>
    <td align="center">31.31/0.8570/0.2381/0.1268</td>
    <td align="center">27.17/0.7948/0.2548/0.1470</td>
    <td align="center">28.44/0.8493/0.1580/0.1394</td>
  </tr>
</table>
</div>

**Comparisons with representative/SoTA ASR models (PSNR/SSIM are tested on Y channel of Ycbcr space).**

We provide three versions of GSASR:
 - Paper: the results we reported in our paper. (not reported) means results are not shown in our paper due to limited pages.
 - Enhanced: we introduce [Rotary Position Embedding (ROPE)](https://github.com/naver-ai/rope-vit) with Flash Attention, and utilize Automatic Mixed Precision (AMP) strategy during training/inference to to reduce memory and time cost.
 - Ultra Performance: based on `Enhanced` settings, we explore the performance upper bound of GSASR by introducing  [HAT-L](https://github.com/XPixelGroup/HAT) encoder and [SA1B](https://ai.meta.com/datasets/segment-anything/) dataset.

## Pre-trained Models (Enhanced and Ultra Performance Version)

| Model Backbone | Training Dataset | Download                                                                                                                                      | Version           |
| :--------------- | :--------------- | :-------------------------------------------------------------------------------------------------------------------------------------------- | :---------------- |
| EDSR           | DIV2K            | [Google Drive](https://drive.google.com/drive/folders/1R6ZCdAd6t_2CCpjCK67F9nag9jitMhI6?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/EDSR_DIV2K) | Enhanced          |
| EDSR           | DF2K             | [Google Drive](https://drive.google.com/drive/folders/16TV2yJt_lfNqJnATtJnEkHV1KoBuW8ww?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/EDSR_DF2K)   | Enhanced          |
| RDN            | DIV2K            | [Google Drive](https://drive.google.com/drive/folders/1guSg28c8gvrTkCvTmNbzqf9vWJfLv58Q?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/RDN_DIV2K) | Enhanced          |
| RDN            | DF2K             | [Google Drive](https://drive.google.com/drive/folders/1vkBvsiiNqTFKmPtNjPlqMn_mh_ClUrKE?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/RDN_DF2K)   | Enhanced          |
| SWIN           | DIV2K            | [Google Drive](https://drive.google.com/drive/folders/1kVLkOs4KrXlXsPsh0oqvey2dvT6TxqH-?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/SWIN_DIV2K) | Enhanced          |
| SWIN           | DF2K             | [Google Drive](https://drive.google.com/drive/folders/1ql6dktVUlQFIoPSJkEuvvMPz9TlacMdy?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/SWIN_DF2K)   | Enhanced          |
| HATL           | SA1B             | [Google Drive](https://drive.google.com/drive/folders/1Pn-4JWvlMj50CulmAcBI1Hssiu-6nSYI?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/HATL-SA1B)   | Ultra Performance |

## Usage

### Preparation
- Pytorch == 2.0 (PyTorch Version must >= 2.0)
- Anaconda
- CUDA Toolkit (necessary)

Firstly, please make sure you have installed [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit-archive)! Since we have hand-crafted CUDA operators, you need to compile them when you run GSASR.

```bash
git clone https://github.com/ChrisDud0257/GSASR
cd GSASR
conda create --name gsasr python=3.10
conda activate gsasr
export CUDA_HOME=${path_to_CUDA} ### specify the path to cuda-11.8
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
python setup_gscuda.py install # gscuda
cd TrainTestGSASR
pip install -r requirements.txt
BASICSR_EXT=True python setup.py develop # basicsr
```

We have tested that the versions of CUDA from 11.0 to 12.4 are all OK.

### Running Inference

You need to properly authenticate with Hugging Face to download our model weights. Once set up, our code will handle it automatically at your first run. You can authenticate by running:

```bash
# This will prompt you to enter your Hugging Face credentials.
huggingface-cli login
```

You can try GSASR easily by launching the Gradio demo or running in command.

### 🚀 Gradio demo
```bash
python demo_gr.py
```

### 💻 CLI
```bash
python inference_enhenced.py \
    --input_img_path <path_to_img> \
    --save_sr_path <path_to_saved_folder> \
    --model <{EDSR_DIV2K, EDSR_DF2K, RDN_DIV2K, RDN_DF2K, SWIN_DIV2K,SWIN_DF2K, HATL_SA1B}> \
    --scale <scale> [--tile_process] [--AMP_test]
```
If it fails to access Huggingface, try to manually download pretrained models and specify local path with `--model_path <path_to_model_weight>`. 

Using `--tile_process` and `--AMP_test` if memory is limited.

## Citation

If you find this research helpful for you, please cite our paper.
```bibtex
@article{chen2025generalized,
  title={Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution},
  author={Chen, Du and Chen, Liyi and Zhang, Zhengqiang and Zhang, Lei},
  journal={arXiv preprint arXiv:2501.06838},
  year={2025}
}
```

## Acknowledgement

This project is built mainly based on the excellent [BasicSR](https://github.com/XPixelGroup/BasicSR), [HAT](https://github.com/XPixelGroup/HAT) and [ROPE-ViT](https://github.com/naver-ai/rope-vit) codeframe. We appreciate it a lot for their developers.

We sincerely thank [Mr.Zhengqiang Zhang](https://github.com/xtudbxk) for his support in the CUDA operator of rasterization.