File size: 13,005 Bytes
de43fde |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 |
---
license: apache-2.0
pipeline_tag: image-to-image
library_name: pytorch
tags:
- super-resolution
- image-enhancement
- generative-modeling
---
# Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution (GSASR)
This repository contains the official models and code for **GSASR**, a novel method for arbitrary-scale super-resolution, presented in the paper [Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution](https://huggingface.co/papers/2501.06838).
📚 [Paper](https://huggingface.co/papers/2501.06838) | 🌐 [Project Page](https://mt-cly.github.io/GSASR.github.io/) | 💻 [Code](https://github.com/ChrisDud0257/GSASR) | 🚀 [Gradio Demo](https://huggingface.co/spaces/mutou0308/GSASR)
## Abstract
Implicit Neural Representations (INR) have been successfully employed for Arbitrary-scale Super-Resolution (ASR). However, INR-based models need to query the multi-layer perceptron module numerous times and render a pixel in each query, resulting in insufficient representation capability and low computational efficiency. Recently, Gaussian Splatting (GS) has shown its advantages over INR in both visual quality and rendering speed in 3D tasks, which motivates us to explore whether GS can be employed for the ASR task. However, directly applying GS to ASR is exceptionally challenging because the original GS is an optimization-based method through overfitting each single scene, while in ASR we aim to learn a single model that can generalize to different images and scaling factors. We overcome these challenges by developing two novel techniques. Firstly, to generalize GS for ASR, we elaborately design an architecture to predict the corresponding image-conditioned Gaussians of the input low-resolution image in a feed-forward manner. Each Gaussian can fit the shape and direction of an area of complex textures, showing powerful representation capability. Secondly, we implement an efficient differentiable 2D GPU/CUDA-based scale-aware rasterization to render super-resolved images by sampling discrete RGB values from the predicted continuous Gaussians. Via end-to-end training, our optimized network, namely GSASR, can perform ASR for any image and unseen scaling factors. Extensive experiments validate the effectiveness of our proposed method.
## Overview
GSASR achieves state-of-the-art in arbitrary-scale super-resolution by representing given low-resolution images as millions of continuous 2D Gaussians.
## Performance
**Comparisons with representative/SoTA ASR models (PSNR/SSIM are tested on Y channel of Ycbcr space).**
<div style="overflow-x:auto; font-size:10px;">
<table>
<tr>
<th rowspan="2">Encoder Backbone</th>
<th rowspan="2">Methods</th>
<th rowspan="2">Version</th>
<th rowspan="2">Training Dataset</th>
<th colspan="3" align="center">PSNR/SSIM/LPIPS/DIST (x4 scaling factor)</th>
</tr>
<tr>
<td align="center">DIV2K</td>
<td align="center">LSDIR</td>
<td align="center">Urban100</td>
</tr>
<!-- EDSR Backbone -->
<tr>
<td rowspan="6">EDSR</td>
<td>LIIF</td>
<td>Paper</td>
<td>DIV2K</td>
<td align="center">30.43/0.8388/0.2662/0.1403</td>
<td align="center">26.21/0.7614/0.2978/0.1678</td>
<td align="center">26.14/0.7885/0.2271/0.1738</td>
</tr>
<tr>
<td>GaussianSR</td>
<td>Paper</td>
<td>DIV2K</td>
<td align="center">30.46/0.8389/0.2684/0.1406</td>
<td align="center">26.23/0.7615/0.3007/0.1679</td>
<td align="center">26.19/0.7893/0.2283/0.1730</td>
</tr>
<tr>
<td>CiaoSR</td>
<td>Paper</td>
<td>DIV2K</td>
<td align="center">30.67/0.8431/0.2585/0.1370</td>
<td align="center">26.42/0.7681/0.2865/0.1631</td>
<td align="center">26.69/0.8091/0.2078/0.1659</td>
</tr>
<tr>
<td>GSASR</td>
<td>Paper Reported</td>
<td>DIV2K</td>
<td align="center">30.89/0.8486/0.2518/0.1301</td>
<td align="center">26.65/0.7774/0.2777/0.1554</td>
<td align="center">27.01/0.8142/0.1987/0.1552</td>
</tr>
<tr>
<td>GSASR</td>
<td>Enhanced</td>
<td>DIV2K</td>
<td align="center">31.01/0.8509/0.2508/0.1306</td>
<td align="center">26.78/0.7813/0.2962/0.1543</td>
<td align="center">27.34/0.8230/0.1920/0.1515</td>
</tr>
<tr>
<td>GSASR</td>
<td>Enhanced</td>
<td>DF2K</td>
<td align="center">31.04/0.8515/0.2512/0.1307</td>
<td align="center">26.82/0.7827/0.2751/0.1540</td>
<td align="center">27.45/0.8256/0.1902/0.1507</td>
</tr>
<!-- RDN Backbone -->
<tr>
<td rowspan="6">RDN</td>
<td>LIIF</td>
<td>Paper</td>
<td>DIV2K</td>
<td align="center">30.71/0.8449/0.2566/0.1354</td>
<td align="center">26.48/0.7714/0.2838/0.1603</td>
<td align="center">26.71/0.8055/0.2062/0.1562</td>
</tr>
<tr>
<td>GaussianSR</td>
<td>Paper</td>
<td>DIV2K</td>
<td align="center">30.76/0.8457/0.2570/0.1347</td>
<td align="center">26.53/0.7727/0.2837/0.1595</td>
<td align="center">26.77/0.8064/0.2069/0.1610</td>
</tr>
<tr>
<td>CiaoSR</td>
<td>Paper</td>
<td>DIV2K</td>
<td align="center">30.91/0.8481/0.2525/0.1327</td>
<td align="center">26.66/0.7770/0.2768/0.1563</td>
<td align="center">27.10/0.8142/0.1966/0.1559</td>
</tr>
<tr>
<td>GSASR</td>
<td>Paper Reported</td>
<td>DIV2K</td>
<td align="center">30.96/0.8500/0.2505/0.1288</td>
<td align="center">26.73/0.7801/0.2752/0.1533</td>
<td align="center">27.15/0.8177/0.1953/0.1515</td>
</tr>
<tr>
<td>GSASR</td>
<td>Enhanced</td>
<td>DIV2K</td>
<td align="center">31.03/0.8513/0.2499/0.1306</td>
<td align="center">26.79/0.7819/0.2740/0.1543</td>
<td align="center">27.37/0.8238/0.1898/0.1511</td>
</tr>
<tr>
<td>GSASR</td>
<td>Enhanced</td>
<td>DF2K</td>
<td align="center">31.10/0.8525/0.2482/0.1296</td>
<td align="center">26.88/0.7848/0.2709/0.1527</td>
<td align="center">27.58/0.8289/0.1849/0.1500</td>
</tr>
<!-- SWIN Backbone -->
<tr>
<td rowspan="4">SWIN</td>
<td>CiaoSR</td>
<td>Paper</td>
<td>DIV2K</td>
<td align="center">31.05/0.8511/0.2487/0.1316</td>
<td align="center">26.80/0.7812/0.2724/0.1552</td>
<td align="center">27.40/0.8231/0.1869/0.1535</td>
</tr>
<tr>
<td>GSASR</td>
<td>Paper (not Reported)</td>
<td>DIV2K</td>
<td align="center">31.06/0.8521/0.2487/0.1270</td>
<td align="center">26.84/0.7837/0.2719/0.1503</td>
<td align="center">27.39/0.8247/0.1913/0.1466</td>
</tr>
<tr>
<td>GSASR</td>
<td>Enhanced</td>
<td>DIV2K</td>
<td align="center">31.10/0.8530/0.2463/0.1285</td>
<td align="center">26.88/0.7849/0.2690/0.1517</td>
<td align="center">27.55/0.8280/0.1850/0.1475</td>
</tr>
<tr>
<td>GSASR</td>
<td>Enhanced</td>
<td>DF2K</td>
<td align="center">31.17/0.8541/0.2456/0.1288</td>
<td align="center">26.96/0.7876/0.2665/0.1513</td>
<td align="center">27.81/0.8343/0.1781/0.1465</td>
</tr>
<!-- HATL Backbone -->
<tr>
<td>HATL</td>
<td>GSASR</td>
<td>Ultra Performance</td>
<td>SA1B</td>
<td align="center">31.31/0.8570/0.2381/0.1268</td>
<td align="center">27.17/0.7948/0.2548/0.1470</td>
<td align="center">28.44/0.8493/0.1580/0.1394</td>
</tr>
</table>
</div>
**Comparisons with representative/SoTA ASR models (PSNR/SSIM are tested on Y channel of Ycbcr space).**
We provide three versions of GSASR:
- Paper: the results we reported in our paper. (not reported) means results are not shown in our paper due to limited pages.
- Enhanced: we introduce [Rotary Position Embedding (ROPE)](https://github.com/naver-ai/rope-vit) with Flash Attention, and utilize Automatic Mixed Precision (AMP) strategy during training/inference to to reduce memory and time cost.
- Ultra Performance: based on `Enhanced` settings, we explore the performance upper bound of GSASR by introducing [HAT-L](https://github.com/XPixelGroup/HAT) encoder and [SA1B](https://ai.meta.com/datasets/segment-anything/) dataset.
## Pre-trained Models (Enhanced and Ultra Performance Version)
| Model Backbone | Training Dataset | Download | Version |
| :--------------- | :--------------- | :-------------------------------------------------------------------------------------------------------------------------------------------- | :---------------- |
| EDSR | DIV2K | [Google Drive](https://drive.google.com/drive/folders/1R6ZCdAd6t_2CCpjCK67F9nag9jitMhI6?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/EDSR_DIV2K) | Enhanced |
| EDSR | DF2K | [Google Drive](https://drive.google.com/drive/folders/16TV2yJt_lfNqJnATtJnEkHV1KoBuW8ww?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/EDSR_DF2K) | Enhanced |
| RDN | DIV2K | [Google Drive](https://drive.google.com/drive/folders/1guSg28c8gvrTkCvTmNbzqf9vWJfLv58Q?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/RDN_DIV2K) | Enhanced |
| RDN | DF2K | [Google Drive](https://drive.google.com/drive/folders/1vkBvsiiNqTFKmPtNjPlqMn_mh_ClUrKE?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/RDN_DF2K) | Enhanced |
| SWIN | DIV2K | [Google Drive](https://drive.google.com/drive/folders/1kVLkOs4KrXlXsPsh0oqvey2dvT6TxqH-?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/SWIN_DIV2K) | Enhanced |
| SWIN | DF2K | [Google Drive](https://drive.google.com/drive/folders/1ql6dktVUlQFIoPSJkEuvvMPz9TlacMdy?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/SWIN_DF2K) | Enhanced |
| HATL | SA1B | [Google Drive](https://drive.google.com/drive/folders/1Pn-4JWvlMj50CulmAcBI1Hssiu-6nSYI?usp=sharing), [Hugging Face](https://huggingface.co/mutou0308/GSASR/tree/main/GSASR_enhenced_ultra/HATL-SA1B) | Ultra Performance |
## Usage
### Preparation
- Pytorch == 2.0 (PyTorch Version must >= 2.0)
- Anaconda
- CUDA Toolkit (necessary)
Firstly, please make sure you have installed [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit-archive)! Since we have hand-crafted CUDA operators, you need to compile them when you run GSASR.
```bash
git clone https://github.com/ChrisDud0257/GSASR
cd GSASR
conda create --name gsasr python=3.10
conda activate gsasr
export CUDA_HOME=${path_to_CUDA} ### specify the path to cuda-11.8
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
python setup_gscuda.py install # gscuda
cd TrainTestGSASR
pip install -r requirements.txt
BASICSR_EXT=True python setup.py develop # basicsr
```
We have tested that the versions of CUDA from 11.0 to 12.4 are all OK.
### Running Inference
You need to properly authenticate with Hugging Face to download our model weights. Once set up, our code will handle it automatically at your first run. You can authenticate by running:
```bash
# This will prompt you to enter your Hugging Face credentials.
huggingface-cli login
```
You can try GSASR easily by launching the Gradio demo or running in command.
### 🚀 Gradio demo
```bash
python demo_gr.py
```
### 💻 CLI
```bash
python inference_enhenced.py \
--input_img_path <path_to_img> \
--save_sr_path <path_to_saved_folder> \
--model <{EDSR_DIV2K, EDSR_DF2K, RDN_DIV2K, RDN_DF2K, SWIN_DIV2K,SWIN_DF2K, HATL_SA1B}> \
--scale <scale> [--tile_process] [--AMP_test]
```
If it fails to access Huggingface, try to manually download pretrained models and specify local path with `--model_path <path_to_model_weight>`.
Using `--tile_process` and `--AMP_test` if memory is limited.
## Citation
If you find this research helpful for you, please cite our paper.
```bibtex
@article{chen2025generalized,
title={Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution},
author={Chen, Du and Chen, Liyi and Zhang, Zhengqiang and Zhang, Lei},
journal={arXiv preprint arXiv:2501.06838},
year={2025}
}
```
## Acknowledgement
This project is built mainly based on the excellent [BasicSR](https://github.com/XPixelGroup/BasicSR), [HAT](https://github.com/XPixelGroup/HAT) and [ROPE-ViT](https://github.com/naver-ai/rope-vit) codeframe. We appreciate it a lot for their developers.
We sincerely thank [Mr.Zhengqiang Zhang](https://github.com/xtudbxk) for his support in the CUDA operator of rasterization. |