Hugging Face's logo

mutou0308
/
GSASR

Model card Files
xet
Community
1
GSASR / README.md
nielsr's picture
nielsr HF Staff
Add comprehensive model card for GSASR
de43fde verified
preview code
|
raw
history blame
13 kB
metadata
license: apache-2.0
pipeline_tag: image-to-image
library_name: pytorch
tags:
  - super-resolution
  - image-enhancement
  - generative-modeling

Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution (GSASR)

This repository contains the official models and code for GSASR, a novel method for arbitrary-scale super-resolution, presented in the paper Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution.

📚 Paper | 🌐 Project Page | 💻 Code | 🚀 Gradio Demo

Abstract

Implicit Neural Representations (INR) have been successfully employed for Arbitrary-scale Super-Resolution (ASR). However, INR-based models need to query the multi-layer perceptron module numerous times and render a pixel in each query, resulting in insufficient representation capability and low computational efficiency. Recently, Gaussian Splatting (GS) has shown its advantages over INR in both visual quality and rendering speed in 3D tasks, which motivates us to explore whether GS can be employed for the ASR task. However, directly applying GS to ASR is exceptionally challenging because the original GS is an optimization-based method through overfitting each single scene, while in ASR we aim to learn a single model that can generalize to different images and scaling factors. We overcome these challenges by developing two novel techniques. Firstly, to generalize GS for ASR, we elaborately design an architecture to predict the corresponding image-conditioned Gaussians of the input low-resolution image in a feed-forward manner. Each Gaussian can fit the shape and direction of an area of complex textures, showing powerful representation capability. Secondly, we implement an efficient differentiable 2D GPU/CUDA-based scale-aware rasterization to render super-resolved images by sampling discrete RGB values from the predicted continuous Gaussians. Via end-to-end training, our optimized network, namely GSASR, can perform ASR for any image and unseen scaling factors. Extensive experiments validate the effectiveness of our proposed method.

Overview

GSASR achieves state-of-the-art in arbitrary-scale super-resolution by representing given low-resolution images as millions of continuous 2D Gaussians.

Fast Rasterization

Performance

Comparisons with representative/SoTA ASR models (PSNR/SSIM are tested on Y channel of Ycbcr space).

Encoder Backbone Methods Version Training Dataset PSNR/SSIM/LPIPS/DIST (x4 scaling factor)
DIV2K LSDIR Urban100
EDSR LIIF Paper DIV2K 30.43/0.8388/0.2662/0.1403 26.21/0.7614/0.2978/0.1678 26.14/0.7885/0.2271/0.1738
GaussianSR Paper DIV2K 30.46/0.8389/0.2684/0.1406 26.23/0.7615/0.3007/0.1679 26.19/0.7893/0.2283/0.1730
CiaoSR Paper DIV2K 30.67/0.8431/0.2585/0.1370 26.42/0.7681/0.2865/0.1631 26.69/0.8091/0.2078/0.1659
GSASR Paper Reported DIV2K 30.89/0.8486/0.2518/0.1301 26.65/0.7774/0.2777/0.1554 27.01/0.8142/0.1987/0.1552
GSASR Enhanced DIV2K 31.01/0.8509/0.2508/0.1306 26.78/0.7813/0.2962/0.1543 27.34/0.8230/0.1920/0.1515
GSASR Enhanced DF2K 31.04/0.8515/0.2512/0.1307 26.82/0.7827/0.2751/0.1540 27.45/0.8256/0.1902/0.1507
RDN LIIF Paper DIV2K 30.71/0.8449/0.2566/0.1354 26.48/0.7714/0.2838/0.1603 26.71/0.8055/0.2062/0.1562
GaussianSR Paper DIV2K 30.76/0.8457/0.2570/0.1347 26.53/0.7727/0.2837/0.1595 26.77/0.8064/0.2069/0.1610
CiaoSR Paper DIV2K 30.91/0.8481/0.2525/0.1327 26.66/0.7770/0.2768/0.1563 27.10/0.8142/0.1966/0.1559
GSASR Paper Reported DIV2K 30.96/0.8500/0.2505/0.1288 26.73/0.7801/0.2752/0.1533 27.15/0.8177/0.1953/0.1515
GSASR Enhanced DIV2K 31.03/0.8513/0.2499/0.1306 26.79/0.7819/0.2740/0.1543 27.37/0.8238/0.1898/0.1511
GSASR Enhanced DF2K 31.10/0.8525/0.2482/0.1296 26.88/0.7848/0.2709/0.1527 27.58/0.8289/0.1849/0.1500
SWIN CiaoSR Paper DIV2K 31.05/0.8511/0.2487/0.1316 26.80/0.7812/0.2724/0.1552 27.40/0.8231/0.1869/0.1535
GSASR Paper (not Reported) DIV2K 31.06/0.8521/0.2487/0.1270 26.84/0.7837/0.2719/0.1503 27.39/0.8247/0.1913/0.1466
GSASR Enhanced DIV2K 31.10/0.8530/0.2463/0.1285 26.88/0.7849/0.2690/0.1517 27.55/0.8280/0.1850/0.1475
GSASR Enhanced DF2K 31.17/0.8541/0.2456/0.1288 26.96/0.7876/0.2665/0.1513 27.81/0.8343/0.1781/0.1465
HATL GSASR Ultra Performance SA1B 31.31/0.8570/0.2381/0.1268 27.17/0.7948/0.2548/0.1470 28.44/0.8493/0.1580/0.1394

Comparisons with representative/SoTA ASR models (PSNR/SSIM are tested on Y channel of Ycbcr space).

We provide three versions of GSASR:

  • Paper: the results we reported in our paper. (not reported) means results are not shown in our paper due to limited pages.
  • Enhanced: we introduce Rotary Position Embedding (ROPE) with Flash Attention, and utilize Automatic Mixed Precision (AMP) strategy during training/inference to to reduce memory and time cost.
  • Ultra Performance: based on Enhanced settings, we explore the performance upper bound of GSASR by introducing HAT-L encoder and SA1B dataset.

Pre-trained Models (Enhanced and Ultra Performance Version)

Model Backbone Training Dataset Download Version
EDSR DIV2K Google Drive, Hugging Face Enhanced
EDSR DF2K Google Drive, Hugging Face Enhanced
RDN DIV2K Google Drive, Hugging Face Enhanced
RDN DF2K Google Drive, Hugging Face Enhanced
SWIN DIV2K Google Drive, Hugging Face Enhanced
SWIN DF2K Google Drive, Hugging Face Enhanced
HATL SA1B Google Drive, Hugging Face Ultra Performance

Usage

Preparation

  • Pytorch == 2.0 (PyTorch Version must >= 2.0)
  • Anaconda
  • CUDA Toolkit (necessary)

Firstly, please make sure you have installed CUDA Toolkit! Since we have hand-crafted CUDA operators, you need to compile them when you run GSASR.

git clone https://github.com/ChrisDud0257/GSASR
cd GSASR
conda create --name gsasr python=3.10
conda activate gsasr
export CUDA_HOME=${path_to_CUDA} ### specify the path to cuda-11.8
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
python setup_gscuda.py install # gscuda
cd TrainTestGSASR
pip install -r requirements.txt
BASICSR_EXT=True python setup.py develop # basicsr

We have tested that the versions of CUDA from 11.0 to 12.4 are all OK.

Running Inference

You need to properly authenticate with Hugging Face to download our model weights. Once set up, our code will handle it automatically at your first run. You can authenticate by running:

# This will prompt you to enter your Hugging Face credentials.
huggingface-cli login

You can try GSASR easily by launching the Gradio demo or running in command.

🚀 Gradio demo 

python demo_gr.py

💻 CLI 

python inference_enhenced.py \
    --input_img_path <path_to_img> \
    --save_sr_path <path_to_saved_folder> \
    --model <{EDSR_DIV2K, EDSR_DF2K, RDN_DIV2K, RDN_DF2K, SWIN_DIV2K,SWIN_DF2K, HATL_SA1B}> \
    --scale <scale> [--tile_process] [--AMP_test]

If it fails to access Huggingface, try to manually download pretrained models and specify local path with --model_path <path_to_model_weight>.

Using --tile_process and --AMP_test if memory is limited.

Citation

If you find this research helpful for you, please cite our paper.

@article{chen2025generalized,
  title={Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution},
  author={Chen, Du and Chen, Liyi and Zhang, Zhengqiang and Zhang, Lei},
  journal={arXiv preprint arXiv:2501.06838},
  year={2025}
}

Acknowledgement

This project is built mainly based on the excellent BasicSR, HAT and ROPE-ViT codeframe. We appreciate it a lot for their developers.

We sincerely thank Mr.Zhengqiang Zhang for his support in the CUDA operator of rasterization.