GB-LSR for arXiv 2606.19617v2
Browse filesModel card rewritten for arXiv v2. The 2,000-step native model gblsr-scalar is replaced by the 1,000,000-step design-study models gblsr-scalar-p32 and gblsr-scalar-p16. The four super-resolution weight files are unchanged. The previous state is tagged arxiv-v1.
- README.md +120 -65
- SHA256SUMS +14 -0
- examples/infer.py +52 -46
- gblsr-scalar-p16/config.json +20 -0
- {gblsr-scalar → gblsr-scalar-p16}/model.safetensors +2 -2
- {gblsr-scalar → gblsr-scalar-p32}/config.json +10 -5
- gblsr-scalar-p32/model.safetensors +3 -0
README.md
CHANGED
|
@@ -9,112 +9,167 @@ tags:
|
|
| 9 |
pipeline_tag: image-to-image
|
| 10 |
metrics:
|
| 11 |
- psnr
|
| 12 |
-
- ssim
|
| 13 |
- lpips
|
| 14 |
---
|
| 15 |
|
| 16 |
-
# GB-LSR:
|
| 17 |
|
| 18 |
[](https://arxiv.org/abs/2606.19617) [](https://doi.org/10.48550/arXiv.2606.19617)
|
| 19 |
|
| 20 |
-
Trained weights for
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
|
|
|
|
|
|
| 26 |
|
| 27 |
-
Code: <https://github.com/KempnerInstitute/gblsr>
|
| 28 |
-
[arXiv:2606.19617](https://arxiv.org/abs/2606.19617).
|
| 29 |
|
| 30 |
## Models
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
## Usage
|
| 44 |
|
| 45 |
-
Install the package
|
| 46 |
|
| 47 |
```bash
|
| 48 |
-
pip install git+https://github.com/KempnerInstitute/gblsr
|
| 49 |
```
|
| 50 |
|
| 51 |
-
|
| 52 |
|
| 53 |
```python
|
| 54 |
-
import torch
|
|
|
|
| 55 |
from safetensors.torch import load_file
|
| 56 |
-
from gblsr import
|
| 57 |
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
recon = out["recon"] # (1, 3, 256, 256) reconstruction
|
| 67 |
```
|
| 68 |
|
| 69 |
-
|
| 70 |
|
| 71 |
```python
|
| 72 |
-
import torch
|
|
|
|
| 73 |
from safetensors.torch import load_file
|
| 74 |
-
from gblsr import
|
| 75 |
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
```
|
| 84 |
|
| 85 |
-
|
|
|
|
| 86 |
|
| 87 |
## Training
|
| 88 |
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
1.44x faster than LIIF-RDN and 3.25x faster than LTE-SwinIR at x4 with
|
| 100 |
-
competitive PSNR-Y; the noLE / nf-variants trade quality for further speed and
|
| 101 |
-
memory. Full tables, baselines, and the scope of the comparison are in the
|
| 102 |
-
paper; these are not state-of-the-art claims over the broader SR literature.
|
| 103 |
|
| 104 |
## License and attribution
|
| 105 |
|
| 106 |
-
Released under
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
|
| 111 |
## Citation
|
| 112 |
|
| 113 |
```bibtex
|
| 114 |
@article{shad2026gblsr,
|
| 115 |
-
title = {GB-LSR:
|
| 116 |
-
|
| 117 |
-
Super-Resolution},
|
| 118 |
author = {Shad, Max and Khoshnevis, Naeem},
|
| 119 |
journal = {arXiv preprint arXiv:2606.19617},
|
| 120 |
year = {2026}
|
|
|
|
| 9 |
pipeline_tag: image-to-image
|
| 10 |
metrics:
|
| 11 |
- psnr
|
|
|
|
| 12 |
- lpips
|
| 13 |
---
|
| 14 |
|
| 15 |
+
# GB-LSR: Local Spectral Decoding with a Learned Global Bandwidth
|
| 16 |
|
| 17 |
[](https://arxiv.org/abs/2606.19617) [](https://doi.org/10.48550/arXiv.2606.19617)
|
| 18 |
|
| 19 |
+
Trained weights for GB-LSR, from [GB-LSR: Local Spectral Decoding with a Learned Global Bandwidth for
|
| 20 |
+
Arbitrary-Scale Super-Resolution](https://arxiv.org/abs/2606.19617). GB-LSR
|
| 21 |
+
partitions the image domain into a fixed grid of non-overlapping square patches. Each patch carries
|
| 22 |
+
coefficients for a truncated Fourier basis, predicted by a single linear projection from shared
|
| 23 |
+
convolutional-encoder features, and one trainable scalar bandwidth is shared across every patch and
|
| 24 |
+
every image. As in earlier local spectral decoders, decoding at a continuous coordinate is a
|
| 25 |
+
fixed-size basis contraction whose cost is set by the spectral cutoff; GB-LSR learns the bandwidth of
|
| 26 |
+
that basis instead of fixing it.
|
| 27 |
|
| 28 |
+
Code: <https://github.com/KempnerInstitute/gblsr>.
|
|
|
|
| 29 |
|
| 30 |
## Models
|
| 31 |
|
| 32 |
+
One checkpoint per configuration, from training seed 0. The paper's tables report means over three
|
| 33 |
+
training seeds; the PSNR of the released checkpoints differs from those means by at most 0.021 dB.
|
| 34 |
+
|
| 35 |
+
### Arbitrary-scale super-resolution
|
| 36 |
+
|
| 37 |
+
RDN encoder, trained on DIV2K at scales from x1 to x4. PSNR-Y (dB) and LPIPS at x4. Learned s: the
|
| 38 |
+
bandwidth after training.
|
| 39 |
+
|
| 40 |
+
| Folder | Model | Params (M) | Set5 | Set14 | B100 | Urban100 | DIV2Kval | Mean LPIPS (lower is better) | Learned s |
|
| 41 |
+
|---|---|---|---|---|---|---|---|---|---|
|
| 42 |
+
| `gblsr-scalar-asr` | GB-LSR-Scalar-ASR (base) | 22.02 | 32.19 | 28.72 | 27.66 | 26.43 | 30.67 | 0.2761 | 0.881 |
|
| 43 |
+
| `gblsr-scalar-asr-noLE` | GB-LSR-Scalar-ASR-noLE | 22.02 | 32.19 | 28.71 | 27.67 | 26.45 | 30.68 | 0.2747 | 0.794 |
|
| 44 |
+
| `gblsr-scalar-asr-nf96-noLE` | GB-LSR-Scalar-ASR-nf96+noLE | 24.93 | 32.23 | 28.74 | 27.68 | 26.47 | 30.70 | 0.2743 | 0.795 |
|
| 45 |
+
| `gblsr-scalar-asr-nf48-noLE` | GB-LSR-Scalar-ASR-nf48+noLE | 20.61 | 32.18 | 28.72 | 27.67 | 26.40 | 30.66 | not scored | 0.784 |
|
| 46 |
+
|
| 47 |
+
noLE: trained and evaluated without the local ensemble. nf96, nf48: RDN encoder with 96 or 48 base
|
| 48 |
+
feature channels instead of 64. Scored as in the paper: PIL bicubic low-resolution inputs; PSNR-Y on
|
| 49 |
+
the BT.601 luminance with a border as wide as the scale removed; Set5, Set14, B100, and Urban100
|
| 50 |
+
ground truth cropped at the bottom and right to a multiple of 12 pixels; DIV2Kval is the DIV2K
|
| 51 |
+
validation split, scored on luminance and so not comparable with published DIV2K numbers;
|
| 52 |
+
LPIPS-AlexNet on the RGB output.
|
| 53 |
+
|
| 54 |
+
Latency at x4 (ms per image) and peak allocated GPU memory, from the paper's timing session on one
|
| 55 |
+
NVIDIA H200 at batch size 1: median of 50 timed passes, averaged over the images and the three seeds'
|
| 56 |
+
models. Speed: the base model's latency divided by the row's, geometric mean over the three test sets.
|
| 57 |
+
|
| 58 |
+
| Folder | Set14 | B100 | Urban100 | Peak memory, Urban100 (MiB) | Speed vs base |
|
| 59 |
+
|---|---|---|---|---|---|
|
| 60 |
+
| `gblsr-scalar-asr` | 34.8 | 25.6 | 109.4 | 899 | 1.00x |
|
| 61 |
+
| `gblsr-scalar-asr-noLE` | 17.7 | 14.6 | 52.5 | 900 | 1.93x |
|
| 62 |
+
| `gblsr-scalar-asr-nf96-noLE` | 20.1 | 16.0 | 55.7 | 1,109 | 1.76x |
|
| 63 |
+
| `gblsr-scalar-asr-nf48-noLE` | 18.8 | 14.4 | 52.9 | 730 | 1.90x |
|
| 64 |
+
|
| 65 |
+
### Design-study models (native reconstruction)
|
| 66 |
+
|
| 67 |
+
GB-LSR-Scalar at patch size 32 and 16, spectral cutoff 16, feature width 128, trained on DTD and DIV2K.
|
| 68 |
+
PSNR-RGB (dB) on images standardized to 256x256 as in the paper: a center crop, after upsampling any
|
| 69 |
+
image with a side under 256. Latent: encoder feature values per image. Learned s is the
|
| 70 |
+
implementation's value, as in the paper's design study (Appendix A.1); the bandwidth of the paper's
|
| 71 |
+
Section 3.1 is this value times P/(P-1) for patch size P. Latency and memory as above, per 256x256
|
| 72 |
+
image. The design study compares these models with other GB-LSR variants only.
|
| 73 |
+
|
| 74 |
+
| Folder | Patch size | Params | Latent | Kodak | Set14 | Urban100 | Learned s | Latency (ms) | Peak memory (MiB) |
|
| 75 |
+
|---|---|---|---|---|---|---|---|---|---|
|
| 76 |
+
| `gblsr-scalar-p32` | 32 | 989,955 | 8,192 | 26.32 | 25.34 | 22.55 | 0.758 | 1.367 | 117.8 |
|
| 77 |
+
| `gblsr-scalar-p16` | 16 | 842,115 | 32,768 | 32.44 | 31.01 | 28.09 | 0.658 | 1.358 | 117.3 |
|
| 78 |
+
|
| 79 |
+
## Results in brief
|
| 80 |
+
|
| 81 |
+
Against the authors' released LIIF, LTE, and SRNO checkpoints on the same RDN encoder, scored under
|
| 82 |
+
one protocol and timed in one session per scale, the base model runs 1.25x faster than LIIF-RDN at x4
|
| 83 |
+
and as fast as SRNO-RDN, and trails the three by 0.07 to 0.79 dB PSNR-Y in distribution. Without the
|
| 84 |
+
local ensemble, PSNR-Y stays within seed variation and the speedup over LIIF-RDN rises to 2.41x at x4
|
| 85 |
+
and 3.00x at x8, at the cost of value jumps at cell boundaries of 0.22 gray levels (of 255) on average
|
| 86 |
+
at x4. The GB-LSR models scored with LPIPS (base, noLE, nf96+noLE) have the highest (worst) mean LPIPS at x4
|
| 87 |
+
of the methods compared. The paper gives the full comparison and its scope.
|
| 88 |
|
| 89 |
## Usage
|
| 90 |
|
| 91 |
+
Install the package at the commit these files were checked with, and the Hub client:
|
| 92 |
|
| 93 |
```bash
|
| 94 |
+
pip install "git+https://github.com/KempnerInstitute/gblsr@6a3cd604c2ba9e868daf01727551d70a76932e1c" huggingface_hub
|
| 95 |
```
|
| 96 |
|
| 97 |
+
Arbitrary-scale super-resolution:
|
| 98 |
|
| 99 |
```python
|
| 100 |
+
import json, torch
|
| 101 |
+
from huggingface_hub import snapshot_download
|
| 102 |
from safetensors.torch import load_file
|
| 103 |
+
from gblsr import GBLSRScalarASR
|
| 104 |
|
| 105 |
+
root = snapshot_download("KempnerInstituteAI/gblsr", revision="arxiv-v2", allow_patterns="gblsr-scalar-asr/*")
|
| 106 |
+
cfg = json.load(open(f"{root}/gblsr-scalar-asr/config.json"))
|
| 107 |
+
model = GBLSRScalarASR(encoder_cfg=cfg["encoder_cfg"], decoder_cfg=cfg["decoder_cfg"]).eval()
|
| 108 |
+
model.load_state_dict(load_file(f"{root}/gblsr-scalar-asr/model.safetensors"))
|
| 109 |
+
|
| 110 |
+
lr = torch.rand(1, 3, 64, 64) # RGB in [0, 1]
|
| 111 |
+
with torch.no_grad(): # any output size; tile_q chunks the decoder's queries
|
| 112 |
+
hr = model.predict_full(lr, H_q=256, W_q=256, tile_q=30000).clamp(0, 1)
|
|
|
|
| 113 |
```
|
| 114 |
|
| 115 |
+
Design-study model:
|
| 116 |
|
| 117 |
```python
|
| 118 |
+
import json, torch
|
| 119 |
+
from huggingface_hub import snapshot_download
|
| 120 |
from safetensors.torch import load_file
|
| 121 |
+
from gblsr import BasisConfig, EncoderConfig, ModelConfig, build_model
|
| 122 |
|
| 123 |
+
root = snapshot_download("KempnerInstituteAI/gblsr", revision="arxiv-v2", allow_patterns="gblsr-scalar-p32/*")
|
| 124 |
+
b = json.load(open(f"{root}/gblsr-scalar-p32/config.json"))["build"]
|
| 125 |
+
model = build_model(
|
| 126 |
+
ModelConfig(arm=b["arm"], image_size=b["image_size"], patch_size=b["patch_size"],
|
| 127 |
+
basis=BasisConfig(patch_size=b["patch_size"], p_max=b["p_max"], s_e_range=tuple(b["s_e_range"])),
|
| 128 |
+
encoder=EncoderConfig(d_feat=b["d_feat"])),
|
| 129 |
+
bandwidth_mode=b["bandwidth_mode"], adapt_order=b["adapt_order"]).eval()
|
| 130 |
+
model.load_state_dict(load_file(f"{root}/gblsr-scalar-p32/model.safetensors"))
|
| 131 |
+
|
| 132 |
+
with torch.no_grad():
|
| 133 |
+
recon = model(torch.rand(1, 3, 256, 256))["recon"].clamp(0, 1) # 256x256 RGB
|
| 134 |
```
|
| 135 |
|
| 136 |
+
`examples/infer.py` runs either kind of model on an image file. `SHA256SUMS` gives the sha256 of this
|
| 137 |
+
card, the example, and every model file.
|
| 138 |
|
| 139 |
## Training
|
| 140 |
|
| 141 |
+
Super-resolution: 1,000,000 steps on DIV2K with an L1 loss on 48x48 low-resolution patches; each
|
| 142 |
+
example is a square crop of side round(48k) pixels with k drawn uniformly from [1, 4], downsampled to
|
| 143 |
+
48x48 with PIL bicubic, and the loss is taken on 2,304 of its pixels; batch size 16, random horizontal
|
| 144 |
+
flips and 90 degree rotations, Adam at a learning rate of 1e-4 halved at 200,000, 400,000, 600,000,
|
| 145 |
+
and 800,000 steps. The bandwidth starts at s = 1.0.
|
| 146 |
|
| 147 |
+
Design study: 1,000,000 steps on 256x256 crops from a mixture of DTD and DIV2K, batch size 8, mean
|
| 148 |
+
squared error loss, AdamW (beta = (0.9, 0.95), no weight decay) at a constant learning rate of 2e-4
|
| 149 |
+
with gradient-norm clipping at 1.0. The bandwidth passes through a log-space sigmoid onto [0.25, 2.0]
|
| 150 |
+
and starts at s = 0.707.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
|
| 152 |
## License and attribution
|
| 153 |
|
| 154 |
+
Released under CC BY-NC 4.0. The super-resolution models are trained on
|
| 155 |
+
[DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/), released "for academic research purpose only";
|
| 156 |
+
the design-study models are trained on [DTD](https://www.robots.ox.ac.uk/~vgg/data/dtd/), released
|
| 157 |
+
for research purposes, and DIV2K. Use of the weights must respect those terms. The `gblsr` code is
|
| 158 |
+
BSD-3-Clause.
|
| 159 |
+
|
| 160 |
+
## Versions
|
| 161 |
+
|
| 162 |
+
- arXiv v2 (this version, tag `arxiv-v2`): the 2,000-step native model `gblsr-scalar` is replaced by
|
| 163 |
+
the 1,000,000-step design-study models `gblsr-scalar-p32` and `gblsr-scalar-p16`; the four
|
| 164 |
+
super-resolution weight files are unchanged.
|
| 165 |
+
- arXiv v1: tag `arxiv-v1`.
|
| 166 |
|
| 167 |
## Citation
|
| 168 |
|
| 169 |
```bibtex
|
| 170 |
@article{shad2026gblsr,
|
| 171 |
+
title = {GB-LSR: Local Spectral Decoding with a Learned Global Bandwidth for
|
| 172 |
+
Arbitrary-Scale Super-Resolution},
|
|
|
|
| 173 |
author = {Shad, Max and Khoshnevis, Naeem},
|
| 174 |
journal = {arXiv preprint arXiv:2606.19617},
|
| 175 |
year = {2026}
|
SHA256SUMS
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
142b04ca87a5360aaf5893f056da90cfb57ab40ee7aac315dc83f6d365c984b6 README.md
|
| 2 |
+
6142037ef2ff5513e50f1f1b932a599567d806d453cfcb6cf6e426d112e9f9c6 examples/infer.py
|
| 3 |
+
e9cd0ef30dd4191612be7505b781e31feed4c566b14837c1b07884deb3527969 gblsr-scalar-asr-nf48-noLE/config.json
|
| 4 |
+
9a9d58e160b76175baa08759acc36d038a2998ad57e11de6c03b798e8ce93d39 gblsr-scalar-asr-nf48-noLE/model.safetensors
|
| 5 |
+
ed1d712dcbaf18740f7a1f5bbd25b6958cd61e306e92a1f9a8871057182ffd93 gblsr-scalar-asr-nf96-noLE/config.json
|
| 6 |
+
69622e7c09d2d5c4ffa8b6049a76df1bc73ee2b0eb2054d3a34269cada3f27f3 gblsr-scalar-asr-nf96-noLE/model.safetensors
|
| 7 |
+
6312e78b9434acb64b6a557154faf091a15984617bf720f67fa2216b8886e2a7 gblsr-scalar-asr-noLE/config.json
|
| 8 |
+
fbeb53249c02e19b7efb6aca69fda2e8879efa21c9ab75a8da249169cd08f385 gblsr-scalar-asr-noLE/model.safetensors
|
| 9 |
+
250631bc86df05e8bea0a69705d84d300ca68d99c584fcc9f1a43b2b5ef3dcde gblsr-scalar-asr/config.json
|
| 10 |
+
dc53b31f15507e4a92089e40d1f0f323390090048d0d41b846a14b9f14c0373a gblsr-scalar-asr/model.safetensors
|
| 11 |
+
d9268a203994cd6ea8e3da7d005cbce35898389e3a0e30441f3da09c66173933 gblsr-scalar-p16/config.json
|
| 12 |
+
40b6ed483353a731329ddce156cccbaee1e750c18b564280ae4b54b43e72b297 gblsr-scalar-p16/model.safetensors
|
| 13 |
+
91f8cab3de49fad27a204f073ea467b37610c81a1a41bb0b6b973fe198a69a4d gblsr-scalar-p32/config.json
|
| 14 |
+
6b2269b67cc0bb81b5d2111f2e7573b936ef914486a9fd9e5e20b758ff7776d6 gblsr-scalar-p32/model.safetensors
|
examples/infer.py
CHANGED
|
@@ -1,14 +1,12 @@
|
|
| 1 |
-
"""
|
| 2 |
|
| 3 |
-
|
|
|
|
| 4 |
|
| 5 |
-
#
|
| 6 |
-
python infer.py --model gblsr-scalar-
|
| 7 |
|
| 8 |
-
|
| 9 |
-
python infer.py --model gblsr-scalar --image in256.png --out recon.png
|
| 10 |
-
|
| 11 |
-
With no --image, runs on a random tensor and just prints the output shape.
|
| 12 |
"""
|
| 13 |
|
| 14 |
from __future__ import annotations
|
|
@@ -17,71 +15,79 @@ import argparse
|
|
| 17 |
import json
|
| 18 |
from pathlib import Path
|
| 19 |
|
|
|
|
| 20 |
import torch
|
|
|
|
| 21 |
from safetensors.torch import load_file
|
| 22 |
|
| 23 |
-
|
| 24 |
|
| 25 |
|
| 26 |
def load_model(name: str):
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
if
|
| 30 |
from gblsr import BasisConfig, EncoderConfig, ModelConfig, build_model
|
| 31 |
|
|
|
|
| 32 |
model = build_model(
|
| 33 |
-
ModelConfig(arm="
|
| 34 |
-
basis=BasisConfig(patch_size=
|
| 35 |
-
|
| 36 |
-
|
|
|
|
| 37 |
else:
|
| 38 |
from gblsr import GBLSRScalarASR
|
| 39 |
|
| 40 |
model = GBLSRScalarASR(encoder_cfg=cfg["encoder_cfg"], decoder_cfg=cfg["decoder_cfg"])
|
| 41 |
-
model.load_state_dict(
|
| 42 |
return model.eval(), cfg
|
| 43 |
|
| 44 |
|
| 45 |
-
def
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
ap = argparse.ArgumentParser()
|
| 47 |
-
ap.add_argument("--model", required=True, help="release folder
|
| 48 |
-
ap.add_argument("--image",
|
| 49 |
-
ap.add_argument("--scale", type=float, default=4.0, help="
|
| 50 |
-
ap.add_argument("--device", default=
|
| 51 |
-
ap.add_argument("--out",
|
| 52 |
args = ap.parse_args()
|
| 53 |
|
| 54 |
-
device = args.device or ("cuda" if torch.cuda.is_available() else "cpu")
|
| 55 |
model, cfg = load_model(args.model)
|
| 56 |
-
model = model.to(device)
|
| 57 |
-
|
| 58 |
-
|
| 59 |
if args.image:
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
x = torch.from_numpy(arr).permute(2, 0, 1).unsqueeze(0)
|
| 65 |
else:
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
with torch.no_grad():
|
| 70 |
-
if
|
| 71 |
-
|
| 72 |
-
out = model.predict_full(x, H_q=H, W_q=W)
|
| 73 |
else:
|
| 74 |
-
|
| 75 |
-
|
| 76 |
out = out.clamp(0, 1)
|
| 77 |
print(f"{args.model}: input {tuple(x.shape)} -> output {tuple(out.shape)}")
|
| 78 |
-
|
| 79 |
if args.out:
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
img = (out[0].permute(1, 2, 0).cpu().numpy() * 255).round().astype("uint8")
|
| 84 |
-
Image.fromarray(img).save(args.out)
|
| 85 |
print(f"wrote {args.out}")
|
| 86 |
|
| 87 |
|
|
|
|
| 1 |
+
"""Run a released GB-LSR model on an image.
|
| 2 |
|
| 3 |
+
# arbitrary-scale super-resolution, x4
|
| 4 |
+
python examples/infer.py --model gblsr-scalar-asr --image in.png --scale 4 --out sr.png
|
| 5 |
|
| 6 |
+
# design-study model; the image is first brought to 256x256 as in the paper
|
| 7 |
+
python examples/infer.py --model gblsr-scalar-p32 --image in.png --out recon.png
|
| 8 |
|
| 9 |
+
Without --image it runs on a random input and prints the output shape.
|
|
|
|
|
|
|
|
|
|
| 10 |
"""
|
| 11 |
|
| 12 |
from __future__ import annotations
|
|
|
|
| 15 |
import json
|
| 16 |
from pathlib import Path
|
| 17 |
|
| 18 |
+
import numpy as np
|
| 19 |
import torch
|
| 20 |
+
from PIL import Image
|
| 21 |
from safetensors.torch import load_file
|
| 22 |
|
| 23 |
+
ROOT = Path(__file__).absolute().parent.parent
|
| 24 |
|
| 25 |
|
| 26 |
def load_model(name: str):
|
| 27 |
+
"""Build the model named by a release folder from its config.json and load its weights."""
|
| 28 |
+
cfg = json.loads((ROOT / name / "config.json").read_text())
|
| 29 |
+
if "build" in cfg:
|
| 30 |
from gblsr import BasisConfig, EncoderConfig, ModelConfig, build_model
|
| 31 |
|
| 32 |
+
b = cfg["build"]
|
| 33 |
model = build_model(
|
| 34 |
+
ModelConfig(arm=b["arm"], image_size=b["image_size"], patch_size=b["patch_size"],
|
| 35 |
+
basis=BasisConfig(patch_size=b["patch_size"], p_max=b["p_max"],
|
| 36 |
+
s_e_range=tuple(b["s_e_range"])),
|
| 37 |
+
encoder=EncoderConfig(d_feat=b["d_feat"])),
|
| 38 |
+
bandwidth_mode=b["bandwidth_mode"], adapt_order=b["adapt_order"])
|
| 39 |
else:
|
| 40 |
from gblsr import GBLSRScalarASR
|
| 41 |
|
| 42 |
model = GBLSRScalarASR(encoder_cfg=cfg["encoder_cfg"], decoder_cfg=cfg["decoder_cfg"])
|
| 43 |
+
model.load_state_dict(load_file(str(ROOT / name / "model.safetensors")), strict=True)
|
| 44 |
return model.eval(), cfg
|
| 45 |
|
| 46 |
|
| 47 |
+
def standardize(img: Image.Image, size: int) -> Image.Image:
|
| 48 |
+
"""Bring an image to size x size as the paper's design study does: bicubic upsampling when a
|
| 49 |
+
side is shorter than size, then a center crop."""
|
| 50 |
+
w, h = img.size
|
| 51 |
+
if w < size or h < size:
|
| 52 |
+
s = max(size / w, size / h)
|
| 53 |
+
img = img.resize((max(int(w * s) + 1, size), max(int(h * s) + 1, size)), Image.BICUBIC)
|
| 54 |
+
w, h = img.size
|
| 55 |
+
x, y = (w - size) // 2, (h - size) // 2
|
| 56 |
+
return img.crop((x, y, x + size, y + size))
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def main() -> None:
|
| 60 |
ap = argparse.ArgumentParser()
|
| 61 |
+
ap.add_argument("--model", required=True, help="release folder, e.g. gblsr-scalar-asr")
|
| 62 |
+
ap.add_argument("--image", help="input image; a random input when omitted")
|
| 63 |
+
ap.add_argument("--scale", type=float, default=4.0, help="upscaling factor for super-resolution models")
|
| 64 |
+
ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
|
| 65 |
+
ap.add_argument("--out", help="where to write the output image")
|
| 66 |
args = ap.parse_args()
|
| 67 |
|
|
|
|
| 68 |
model, cfg = load_model(args.model)
|
| 69 |
+
model = model.to(args.device)
|
| 70 |
+
native = "build" in cfg
|
|
|
|
| 71 |
if args.image:
|
| 72 |
+
img = Image.open(args.image).convert("RGB")
|
| 73 |
+
if native:
|
| 74 |
+
img = standardize(img, cfg["build"]["image_size"])
|
| 75 |
+
x = torch.from_numpy(np.asarray(img, dtype=np.float32) / 255.0).permute(2, 0, 1).unsqueeze(0)
|
|
|
|
| 76 |
else:
|
| 77 |
+
size = cfg["build"]["image_size"] if native else 64
|
| 78 |
+
x = torch.rand(1, 3, size, size)
|
| 79 |
+
x = x.to(args.device)
|
| 80 |
with torch.no_grad():
|
| 81 |
+
if native:
|
| 82 |
+
out = model(x)["recon"]
|
|
|
|
| 83 |
else:
|
| 84 |
+
out = model.predict_full(x, H_q=round(x.shape[-2] * args.scale), W_q=round(x.shape[-1] * args.scale),
|
| 85 |
+
tile_q=30000)
|
| 86 |
out = out.clamp(0, 1)
|
| 87 |
print(f"{args.model}: input {tuple(x.shape)} -> output {tuple(out.shape)}")
|
|
|
|
| 88 |
if args.out:
|
| 89 |
+
arr = (out[0].permute(1, 2, 0).cpu().numpy() * 255).round().astype(np.uint8)
|
| 90 |
+
Image.fromarray(arr).save(args.out)
|
|
|
|
|
|
|
|
|
|
| 91 |
print(f"wrote {args.out}")
|
| 92 |
|
| 93 |
|
gblsr-scalar-p16/config.json
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "gblsr-scalar-p16",
|
| 3 |
+
"build": {
|
| 4 |
+
"arm": "local_spectral",
|
| 5 |
+
"image_size": 256,
|
| 6 |
+
"patch_size": 16,
|
| 7 |
+
"p_max": 16,
|
| 8 |
+
"d_feat": 128,
|
| 9 |
+
"s_e_range": [
|
| 10 |
+
0.25,
|
| 11 |
+
2.0
|
| 12 |
+
],
|
| 13 |
+
"bandwidth_mode": "global_scalar",
|
| 14 |
+
"adapt_order": false
|
| 15 |
+
},
|
| 16 |
+
"seed": 0,
|
| 17 |
+
"trainable_params": 842115,
|
| 18 |
+
"trained_steps": 1000000,
|
| 19 |
+
"learned_bandwidth": 0.658
|
| 20 |
+
}
|
{gblsr-scalar → gblsr-scalar-p16}/model.safetensors
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:40b6ed483353a731329ddce156cccbaee1e750c18b564280ae4b54b43e72b297
|
| 3 |
+
size 3370836
|
{gblsr-scalar → gblsr-scalar-p32}/config.json
RENAMED
|
@@ -1,15 +1,20 @@
|
|
| 1 |
{
|
| 2 |
-
"model": "gblsr-scalar",
|
| 3 |
"build": {
|
| 4 |
"arm": "local_spectral",
|
|
|
|
| 5 |
"patch_size": 32,
|
| 6 |
"p_max": 16,
|
| 7 |
-
"bandwidth_mode": "global_scalar",
|
| 8 |
-
"adapt_order": false,
|
| 9 |
"d_feat": 128,
|
| 10 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
},
|
| 12 |
"seed": 0,
|
| 13 |
"trainable_params": 989955,
|
| 14 |
-
"trained_steps":
|
|
|
|
| 15 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"model": "gblsr-scalar-p32",
|
| 3 |
"build": {
|
| 4 |
"arm": "local_spectral",
|
| 5 |
+
"image_size": 256,
|
| 6 |
"patch_size": 32,
|
| 7 |
"p_max": 16,
|
|
|
|
|
|
|
| 8 |
"d_feat": 128,
|
| 9 |
+
"s_e_range": [
|
| 10 |
+
0.25,
|
| 11 |
+
2.0
|
| 12 |
+
],
|
| 13 |
+
"bandwidth_mode": "global_scalar",
|
| 14 |
+
"adapt_order": false
|
| 15 |
},
|
| 16 |
"seed": 0,
|
| 17 |
"trainable_params": 989955,
|
| 18 |
+
"trained_steps": 1000000,
|
| 19 |
+
"learned_bandwidth": 0.7585
|
| 20 |
}
|
gblsr-scalar-p32/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6b2269b67cc0bb81b5d2111f2e7573b936ef914486a9fd9e5e20b758ff7776d6
|
| 3 |
+
size 3962548
|