KempnerInstitute commited on
Commit
7b0bd38
·
verified ·
1 Parent(s): d196260

GB-LSR for arXiv 2606.19617v2

Browse files

Model card rewritten for arXiv v2. The 2,000-step native model gblsr-scalar is replaced by the 1,000,000-step design-study models gblsr-scalar-p32 and gblsr-scalar-p16. The four super-resolution weight files are unchanged. The previous state is tagged arxiv-v1.

README.md CHANGED
@@ -9,112 +9,167 @@ tags:
9
  pipeline_tag: image-to-image
10
  metrics:
11
  - psnr
12
- - ssim
13
  - lpips
14
  ---
15
 
16
- # GB-LSR: Global-Bandwidth Local Spectral Representation
17
 
18
  [![arXiv](https://img.shields.io/badge/arXiv-2606.19617-b31b1b.svg)](https://arxiv.org/abs/2606.19617) [![DOI](https://img.shields.io/badge/DOI-10.48550%2FarXiv.2606.19617-blue.svg)](https://doi.org/10.48550/arXiv.2606.19617)
19
 
20
- Trained weights for **GB-LSR**, a fixed-grid local spectral image
21
- representation: the image is partitioned into non-overlapping patches, each
22
- carrying a small block of truncated-Fourier coefficients predicted from a
23
- shared convolutional encoder, with a single trainable scalar bandwidth shared
24
- globally across all patches. Reconstruction at any continuous coordinate is a
25
- fixed-size basis contraction whose cost is independent of image size.
 
 
26
 
27
- Code: <https://github.com/KempnerInstitute/gblsr> · Paper:
28
- [arXiv:2606.19617](https://arxiv.org/abs/2606.19617).
29
 
30
  ## Models
31
 
32
- | Folder | Model | Params | Notes |
33
- |---|---|---|---|
34
- | `gblsr-scalar` | GB-LSR-Scalar (native reconstruction) | 0.99M | main native-benchmark variant |
35
- | `gblsr-scalar-asr` | GB-LSR-Scalar-ASR (base) | 22.02M | arbitrary-scale SR, RDN encoder |
36
- | `gblsr-scalar-asr-noLE` | ASR, no local ensemble | 22.02M | faster; trained + eval'd without 4-corner LE |
37
- | `gblsr-scalar-asr-nf96-noLE` | ASR, wider encoder + noLE | 24.93M | small quality gain |
38
- | `gblsr-scalar-asr-nf48-noLE` | ASR, narrower encoder + noLE | 20.61M | aggressive-efficiency |
39
-
40
- One representative seed (seed 0) is released per model; per-seed variance is
41
- negligible (e.g. ASR PSNR-Y std about 0.004 dB).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
 
43
  ## Usage
44
 
45
- Install the package (provides the architecture), then load the weights:
46
 
47
  ```bash
48
- pip install git+https://github.com/KempnerInstitute/gblsr
49
  ```
50
 
51
- **Native reconstruction (GB-LSR-Scalar):**
52
 
53
  ```python
54
- import torch
 
55
  from safetensors.torch import load_file
56
- from gblsr import build_model, ModelConfig, BasisConfig, EncoderConfig
57
 
58
- model = build_model(
59
- ModelConfig(arm="local_spectral", image_size=256, patch_size=32,
60
- basis=BasisConfig(patch_size=32, p_max=16, s_e_range=(0.25, 2.0)),
61
- encoder=EncoderConfig()),
62
- bandwidth_mode="global_scalar", adapt_order=False).eval()
63
- model.load_state_dict(load_file("gblsr-scalar/model.safetensors"))
64
-
65
- out = model(torch.rand(1, 3, 256, 256)) # dict of outputs
66
- recon = out["recon"] # (1, 3, 256, 256) reconstruction
67
  ```
68
 
69
- **Arbitrary-scale SR (any GB-LSR-Scalar-ASR variant):**
70
 
71
  ```python
72
- import torch
 
73
  from safetensors.torch import load_file
74
- from gblsr import GBLSRScalarASR
75
 
76
- # match encoder_cfg / decoder_cfg to the folder's config.json
77
- model = GBLSRScalarASR(encoder_cfg={"num_features": 96},
78
- decoder_cfg={"local_ensemble": False}).eval() # nf96+noLE
79
- model.load_state_dict(load_file("gblsr-scalar-asr-nf96-noLE/model.safetensors"))
80
-
81
- lr = torch.rand(1, 3, 64, 64)
82
- hr = model.predict_full(lr, H_q=256, W_q=256) # (1, 3, 256, 256), any target size
 
 
 
 
83
  ```
84
 
85
- Each model's exact build config is in its `config.json`.
 
86
 
87
  ## Training
88
 
89
- All variants trained on a DTD + DIV2K mixture (native) / DIV2K (ASR), three
90
- seeds, AdamW, pointwise MSE. The ASR models train for 1,000,000 steps. See the
91
- paper's appendix for the full protocol.
92
-
93
- ## Evaluation (summary)
94
 
95
- Under a fixed matched-budget protocol, GB-LSR-Scalar outperforms matched-budget
96
- amortized LIIF / LTE / WIRE re-implementations on the standardized
97
- 256x256 native-reconstruction benchmark while running at roughly one-quarter of
98
- the slowest baseline's inference cost. The arbitrary-scale base model runs
99
- 1.44x faster than LIIF-RDN and 3.25x faster than LTE-SwinIR at x4 with
100
- competitive PSNR-Y; the noLE / nf-variants trade quality for further speed and
101
- memory. Full tables, baselines, and the scope of the comparison are in the
102
- paper; these are not state-of-the-art claims over the broader SR literature.
103
 
104
  ## License and attribution
105
 
106
- Released under **CC BY-NC 4.0 (non-commercial)**. The weights are derived from
107
- training on **DIV2K** ("for academic research purpose only") and **DTD**;
108
- downstream use must respect those source-dataset terms. The `gblsr` code is
109
- BSD-3-Clause; this non-commercial term applies to the trained weights here.
 
 
 
 
 
 
 
 
110
 
111
  ## Citation
112
 
113
  ```bibtex
114
  @article{shad2026gblsr,
115
- title = {GB-LSR: A Fast Local Spectral Image Representation with a
116
- Single Global Bandwidth for Continuous Reconstruction and
117
- Super-Resolution},
118
  author = {Shad, Max and Khoshnevis, Naeem},
119
  journal = {arXiv preprint arXiv:2606.19617},
120
  year = {2026}
 
9
  pipeline_tag: image-to-image
10
  metrics:
11
  - psnr
 
12
  - lpips
13
  ---
14
 
15
+ # GB-LSR: Local Spectral Decoding with a Learned Global Bandwidth
16
 
17
  [![arXiv](https://img.shields.io/badge/arXiv-2606.19617-b31b1b.svg)](https://arxiv.org/abs/2606.19617) [![DOI](https://img.shields.io/badge/DOI-10.48550%2FarXiv.2606.19617-blue.svg)](https://doi.org/10.48550/arXiv.2606.19617)
18
 
19
+ Trained weights for GB-LSR, from [GB-LSR: Local Spectral Decoding with a Learned Global Bandwidth for
20
+ Arbitrary-Scale Super-Resolution](https://arxiv.org/abs/2606.19617). GB-LSR
21
+ partitions the image domain into a fixed grid of non-overlapping square patches. Each patch carries
22
+ coefficients for a truncated Fourier basis, predicted by a single linear projection from shared
23
+ convolutional-encoder features, and one trainable scalar bandwidth is shared across every patch and
24
+ every image. As in earlier local spectral decoders, decoding at a continuous coordinate is a
25
+ fixed-size basis contraction whose cost is set by the spectral cutoff; GB-LSR learns the bandwidth of
26
+ that basis instead of fixing it.
27
 
28
+ Code: <https://github.com/KempnerInstitute/gblsr>.
 
29
 
30
  ## Models
31
 
32
+ One checkpoint per configuration, from training seed 0. The paper's tables report means over three
33
+ training seeds; the PSNR of the released checkpoints differs from those means by at most 0.021 dB.
34
+
35
+ ### Arbitrary-scale super-resolution
36
+
37
+ RDN encoder, trained on DIV2K at scales from x1 to x4. PSNR-Y (dB) and LPIPS at x4. Learned s: the
38
+ bandwidth after training.
39
+
40
+ | Folder | Model | Params (M) | Set5 | Set14 | B100 | Urban100 | DIV2Kval | Mean LPIPS (lower is better) | Learned s |
41
+ |---|---|---|---|---|---|---|---|---|---|
42
+ | `gblsr-scalar-asr` | GB-LSR-Scalar-ASR (base) | 22.02 | 32.19 | 28.72 | 27.66 | 26.43 | 30.67 | 0.2761 | 0.881 |
43
+ | `gblsr-scalar-asr-noLE` | GB-LSR-Scalar-ASR-noLE | 22.02 | 32.19 | 28.71 | 27.67 | 26.45 | 30.68 | 0.2747 | 0.794 |
44
+ | `gblsr-scalar-asr-nf96-noLE` | GB-LSR-Scalar-ASR-nf96+noLE | 24.93 | 32.23 | 28.74 | 27.68 | 26.47 | 30.70 | 0.2743 | 0.795 |
45
+ | `gblsr-scalar-asr-nf48-noLE` | GB-LSR-Scalar-ASR-nf48+noLE | 20.61 | 32.18 | 28.72 | 27.67 | 26.40 | 30.66 | not scored | 0.784 |
46
+
47
+ noLE: trained and evaluated without the local ensemble. nf96, nf48: RDN encoder with 96 or 48 base
48
+ feature channels instead of 64. Scored as in the paper: PIL bicubic low-resolution inputs; PSNR-Y on
49
+ the BT.601 luminance with a border as wide as the scale removed; Set5, Set14, B100, and Urban100
50
+ ground truth cropped at the bottom and right to a multiple of 12 pixels; DIV2Kval is the DIV2K
51
+ validation split, scored on luminance and so not comparable with published DIV2K numbers;
52
+ LPIPS-AlexNet on the RGB output.
53
+
54
+ Latency at x4 (ms per image) and peak allocated GPU memory, from the paper's timing session on one
55
+ NVIDIA H200 at batch size 1: median of 50 timed passes, averaged over the images and the three seeds'
56
+ models. Speed: the base model's latency divided by the row's, geometric mean over the three test sets.
57
+
58
+ | Folder | Set14 | B100 | Urban100 | Peak memory, Urban100 (MiB) | Speed vs base |
59
+ |---|---|---|---|---|---|
60
+ | `gblsr-scalar-asr` | 34.8 | 25.6 | 109.4 | 899 | 1.00x |
61
+ | `gblsr-scalar-asr-noLE` | 17.7 | 14.6 | 52.5 | 900 | 1.93x |
62
+ | `gblsr-scalar-asr-nf96-noLE` | 20.1 | 16.0 | 55.7 | 1,109 | 1.76x |
63
+ | `gblsr-scalar-asr-nf48-noLE` | 18.8 | 14.4 | 52.9 | 730 | 1.90x |
64
+
65
+ ### Design-study models (native reconstruction)
66
+
67
+ GB-LSR-Scalar at patch size 32 and 16, spectral cutoff 16, feature width 128, trained on DTD and DIV2K.
68
+ PSNR-RGB (dB) on images standardized to 256x256 as in the paper: a center crop, after upsampling any
69
+ image with a side under 256. Latent: encoder feature values per image. Learned s is the
70
+ implementation's value, as in the paper's design study (Appendix A.1); the bandwidth of the paper's
71
+ Section 3.1 is this value times P/(P-1) for patch size P. Latency and memory as above, per 256x256
72
+ image. The design study compares these models with other GB-LSR variants only.
73
+
74
+ | Folder | Patch size | Params | Latent | Kodak | Set14 | Urban100 | Learned s | Latency (ms) | Peak memory (MiB) |
75
+ |---|---|---|---|---|---|---|---|---|---|
76
+ | `gblsr-scalar-p32` | 32 | 989,955 | 8,192 | 26.32 | 25.34 | 22.55 | 0.758 | 1.367 | 117.8 |
77
+ | `gblsr-scalar-p16` | 16 | 842,115 | 32,768 | 32.44 | 31.01 | 28.09 | 0.658 | 1.358 | 117.3 |
78
+
79
+ ## Results in brief
80
+
81
+ Against the authors' released LIIF, LTE, and SRNO checkpoints on the same RDN encoder, scored under
82
+ one protocol and timed in one session per scale, the base model runs 1.25x faster than LIIF-RDN at x4
83
+ and as fast as SRNO-RDN, and trails the three by 0.07 to 0.79 dB PSNR-Y in distribution. Without the
84
+ local ensemble, PSNR-Y stays within seed variation and the speedup over LIIF-RDN rises to 2.41x at x4
85
+ and 3.00x at x8, at the cost of value jumps at cell boundaries of 0.22 gray levels (of 255) on average
86
+ at x4. The GB-LSR models scored with LPIPS (base, noLE, nf96+noLE) have the highest (worst) mean LPIPS at x4
87
+ of the methods compared. The paper gives the full comparison and its scope.
88
 
89
  ## Usage
90
 
91
+ Install the package at the commit these files were checked with, and the Hub client:
92
 
93
  ```bash
94
+ pip install "git+https://github.com/KempnerInstitute/gblsr@6a3cd604c2ba9e868daf01727551d70a76932e1c" huggingface_hub
95
  ```
96
 
97
+ Arbitrary-scale super-resolution:
98
 
99
  ```python
100
+ import json, torch
101
+ from huggingface_hub import snapshot_download
102
  from safetensors.torch import load_file
103
+ from gblsr import GBLSRScalarASR
104
 
105
+ root = snapshot_download("KempnerInstituteAI/gblsr", revision="arxiv-v2", allow_patterns="gblsr-scalar-asr/*")
106
+ cfg = json.load(open(f"{root}/gblsr-scalar-asr/config.json"))
107
+ model = GBLSRScalarASR(encoder_cfg=cfg["encoder_cfg"], decoder_cfg=cfg["decoder_cfg"]).eval()
108
+ model.load_state_dict(load_file(f"{root}/gblsr-scalar-asr/model.safetensors"))
109
+
110
+ lr = torch.rand(1, 3, 64, 64) # RGB in [0, 1]
111
+ with torch.no_grad(): # any output size; tile_q chunks the decoder's queries
112
+ hr = model.predict_full(lr, H_q=256, W_q=256, tile_q=30000).clamp(0, 1)
 
113
  ```
114
 
115
+ Design-study model:
116
 
117
  ```python
118
+ import json, torch
119
+ from huggingface_hub import snapshot_download
120
  from safetensors.torch import load_file
121
+ from gblsr import BasisConfig, EncoderConfig, ModelConfig, build_model
122
 
123
+ root = snapshot_download("KempnerInstituteAI/gblsr", revision="arxiv-v2", allow_patterns="gblsr-scalar-p32/*")
124
+ b = json.load(open(f"{root}/gblsr-scalar-p32/config.json"))["build"]
125
+ model = build_model(
126
+ ModelConfig(arm=b["arm"], image_size=b["image_size"], patch_size=b["patch_size"],
127
+ basis=BasisConfig(patch_size=b["patch_size"], p_max=b["p_max"], s_e_range=tuple(b["s_e_range"])),
128
+ encoder=EncoderConfig(d_feat=b["d_feat"])),
129
+ bandwidth_mode=b["bandwidth_mode"], adapt_order=b["adapt_order"]).eval()
130
+ model.load_state_dict(load_file(f"{root}/gblsr-scalar-p32/model.safetensors"))
131
+
132
+ with torch.no_grad():
133
+ recon = model(torch.rand(1, 3, 256, 256))["recon"].clamp(0, 1) # 256x256 RGB
134
  ```
135
 
136
+ `examples/infer.py` runs either kind of model on an image file. `SHA256SUMS` gives the sha256 of this
137
+ card, the example, and every model file.
138
 
139
  ## Training
140
 
141
+ Super-resolution: 1,000,000 steps on DIV2K with an L1 loss on 48x48 low-resolution patches; each
142
+ example is a square crop of side round(48k) pixels with k drawn uniformly from [1, 4], downsampled to
143
+ 48x48 with PIL bicubic, and the loss is taken on 2,304 of its pixels; batch size 16, random horizontal
144
+ flips and 90 degree rotations, Adam at a learning rate of 1e-4 halved at 200,000, 400,000, 600,000,
145
+ and 800,000 steps. The bandwidth starts at s = 1.0.
146
 
147
+ Design study: 1,000,000 steps on 256x256 crops from a mixture of DTD and DIV2K, batch size 8, mean
148
+ squared error loss, AdamW (beta = (0.9, 0.95), no weight decay) at a constant learning rate of 2e-4
149
+ with gradient-norm clipping at 1.0. The bandwidth passes through a log-space sigmoid onto [0.25, 2.0]
150
+ and starts at s = 0.707.
 
 
 
 
151
 
152
  ## License and attribution
153
 
154
+ Released under CC BY-NC 4.0. The super-resolution models are trained on
155
+ [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/), released "for academic research purpose only";
156
+ the design-study models are trained on [DTD](https://www.robots.ox.ac.uk/~vgg/data/dtd/), released
157
+ for research purposes, and DIV2K. Use of the weights must respect those terms. The `gblsr` code is
158
+ BSD-3-Clause.
159
+
160
+ ## Versions
161
+
162
+ - arXiv v2 (this version, tag `arxiv-v2`): the 2,000-step native model `gblsr-scalar` is replaced by
163
+ the 1,000,000-step design-study models `gblsr-scalar-p32` and `gblsr-scalar-p16`; the four
164
+ super-resolution weight files are unchanged.
165
+ - arXiv v1: tag `arxiv-v1`.
166
 
167
  ## Citation
168
 
169
  ```bibtex
170
  @article{shad2026gblsr,
171
+ title = {GB-LSR: Local Spectral Decoding with a Learned Global Bandwidth for
172
+ Arbitrary-Scale Super-Resolution},
 
173
  author = {Shad, Max and Khoshnevis, Naeem},
174
  journal = {arXiv preprint arXiv:2606.19617},
175
  year = {2026}
SHA256SUMS ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 142b04ca87a5360aaf5893f056da90cfb57ab40ee7aac315dc83f6d365c984b6 README.md
2
+ 6142037ef2ff5513e50f1f1b932a599567d806d453cfcb6cf6e426d112e9f9c6 examples/infer.py
3
+ e9cd0ef30dd4191612be7505b781e31feed4c566b14837c1b07884deb3527969 gblsr-scalar-asr-nf48-noLE/config.json
4
+ 9a9d58e160b76175baa08759acc36d038a2998ad57e11de6c03b798e8ce93d39 gblsr-scalar-asr-nf48-noLE/model.safetensors
5
+ ed1d712dcbaf18740f7a1f5bbd25b6958cd61e306e92a1f9a8871057182ffd93 gblsr-scalar-asr-nf96-noLE/config.json
6
+ 69622e7c09d2d5c4ffa8b6049a76df1bc73ee2b0eb2054d3a34269cada3f27f3 gblsr-scalar-asr-nf96-noLE/model.safetensors
7
+ 6312e78b9434acb64b6a557154faf091a15984617bf720f67fa2216b8886e2a7 gblsr-scalar-asr-noLE/config.json
8
+ fbeb53249c02e19b7efb6aca69fda2e8879efa21c9ab75a8da249169cd08f385 gblsr-scalar-asr-noLE/model.safetensors
9
+ 250631bc86df05e8bea0a69705d84d300ca68d99c584fcc9f1a43b2b5ef3dcde gblsr-scalar-asr/config.json
10
+ dc53b31f15507e4a92089e40d1f0f323390090048d0d41b846a14b9f14c0373a gblsr-scalar-asr/model.safetensors
11
+ d9268a203994cd6ea8e3da7d005cbce35898389e3a0e30441f3da09c66173933 gblsr-scalar-p16/config.json
12
+ 40b6ed483353a731329ddce156cccbaee1e750c18b564280ae4b54b43e72b297 gblsr-scalar-p16/model.safetensors
13
+ 91f8cab3de49fad27a204f073ea467b37610c81a1a41bb0b6b973fe198a69a4d gblsr-scalar-p32/config.json
14
+ 6b2269b67cc0bb81b5d2111f2e7573b936ef914486a9fd9e5e20b758ff7776d6 gblsr-scalar-p32/model.safetensors
examples/infer.py CHANGED
@@ -1,14 +1,12 @@
1
- """Minimal inference example for the GB-LSR Hugging Face release.
2
 
3
- Loads a released model from its safetensors + config.json and runs it.
 
4
 
5
- # arbitrary-scale SR on an image, 4x
6
- python infer.py --model gblsr-scalar-asr --image in.png --scale 4 --out sr.png
7
 
8
- # native reconstruction of a 256x256 image
9
- python infer.py --model gblsr-scalar --image in256.png --out recon.png
10
-
11
- With no --image, runs on a random tensor and just prints the output shape.
12
  """
13
 
14
  from __future__ import annotations
@@ -17,71 +15,79 @@ import argparse
17
  import json
18
  from pathlib import Path
19
 
 
20
  import torch
 
21
  from safetensors.torch import load_file
22
 
23
- HERE = Path(__file__).resolve().parent.parent # release root
24
 
25
 
26
  def load_model(name: str):
27
- cfg = json.loads((HERE / name / "config.json").read_text())
28
- weights = load_file(str(HERE / name / "model.safetensors"))
29
- if name == "gblsr-scalar":
30
  from gblsr import BasisConfig, EncoderConfig, ModelConfig, build_model
31
 
 
32
  model = build_model(
33
- ModelConfig(arm="local_spectral", image_size=256, patch_size=32,
34
- basis=BasisConfig(patch_size=32, p_max=16, s_e_range=(0.25, 2.0)),
35
- encoder=EncoderConfig()),
36
- bandwidth_mode="global_scalar", adapt_order=False)
 
37
  else:
38
  from gblsr import GBLSRScalarASR
39
 
40
  model = GBLSRScalarASR(encoder_cfg=cfg["encoder_cfg"], decoder_cfg=cfg["decoder_cfg"])
41
- model.load_state_dict(weights, strict=True)
42
  return model.eval(), cfg
43
 
44
 
45
- def main():
 
 
 
 
 
 
 
 
 
 
 
 
46
  ap = argparse.ArgumentParser()
47
- ap.add_argument("--model", required=True, help="release folder name, e.g. gblsr-scalar-asr")
48
- ap.add_argument("--image", default=None, help="input image (PNG/JPG); random if omitted")
49
- ap.add_argument("--scale", type=float, default=4.0, help="ASR upscale factor")
50
- ap.add_argument("--device", default=None, help="cuda / cpu (default: cuda if available)")
51
- ap.add_argument("--out", default=None)
52
  args = ap.parse_args()
53
 
54
- device = args.device or ("cuda" if torch.cuda.is_available() else "cpu")
55
  model, cfg = load_model(args.model)
56
- model = model.to(device)
57
- is_asr = args.model != "gblsr-scalar"
58
-
59
  if args.image:
60
- from PIL import Image
61
- import numpy as np
62
-
63
- arr = np.asarray(Image.open(args.image).convert("RGB"), dtype="float32") / 255.0
64
- x = torch.from_numpy(arr).permute(2, 0, 1).unsqueeze(0)
65
  else:
66
- x = torch.rand(1, 3, 64 if is_asr else 256, 64 if is_asr else 256)
67
- x = x.to(device)
68
-
69
  with torch.no_grad():
70
- if is_asr:
71
- H, W = int(x.shape[-2] * args.scale), int(x.shape[-1] * args.scale)
72
- out = model.predict_full(x, H_q=H, W_q=W)
73
  else:
74
- d = model(x)
75
- out = next(v for v in d.values() if torch.is_tensor(v) and v.dim() == 4 and v.shape[1] == 3)
76
  out = out.clamp(0, 1)
77
  print(f"{args.model}: input {tuple(x.shape)} -> output {tuple(out.shape)}")
78
-
79
  if args.out:
80
- from PIL import Image
81
- import numpy as np
82
-
83
- img = (out[0].permute(1, 2, 0).cpu().numpy() * 255).round().astype("uint8")
84
- Image.fromarray(img).save(args.out)
85
  print(f"wrote {args.out}")
86
 
87
 
 
1
+ """Run a released GB-LSR model on an image.
2
 
3
+ # arbitrary-scale super-resolution, x4
4
+ python examples/infer.py --model gblsr-scalar-asr --image in.png --scale 4 --out sr.png
5
 
6
+ # design-study model; the image is first brought to 256x256 as in the paper
7
+ python examples/infer.py --model gblsr-scalar-p32 --image in.png --out recon.png
8
 
9
+ Without --image it runs on a random input and prints the output shape.
 
 
 
10
  """
11
 
12
  from __future__ import annotations
 
15
  import json
16
  from pathlib import Path
17
 
18
+ import numpy as np
19
  import torch
20
+ from PIL import Image
21
  from safetensors.torch import load_file
22
 
23
+ ROOT = Path(__file__).absolute().parent.parent
24
 
25
 
26
  def load_model(name: str):
27
+ """Build the model named by a release folder from its config.json and load its weights."""
28
+ cfg = json.loads((ROOT / name / "config.json").read_text())
29
+ if "build" in cfg:
30
  from gblsr import BasisConfig, EncoderConfig, ModelConfig, build_model
31
 
32
+ b = cfg["build"]
33
  model = build_model(
34
+ ModelConfig(arm=b["arm"], image_size=b["image_size"], patch_size=b["patch_size"],
35
+ basis=BasisConfig(patch_size=b["patch_size"], p_max=b["p_max"],
36
+ s_e_range=tuple(b["s_e_range"])),
37
+ encoder=EncoderConfig(d_feat=b["d_feat"])),
38
+ bandwidth_mode=b["bandwidth_mode"], adapt_order=b["adapt_order"])
39
  else:
40
  from gblsr import GBLSRScalarASR
41
 
42
  model = GBLSRScalarASR(encoder_cfg=cfg["encoder_cfg"], decoder_cfg=cfg["decoder_cfg"])
43
+ model.load_state_dict(load_file(str(ROOT / name / "model.safetensors")), strict=True)
44
  return model.eval(), cfg
45
 
46
 
47
+ def standardize(img: Image.Image, size: int) -> Image.Image:
48
+ """Bring an image to size x size as the paper's design study does: bicubic upsampling when a
49
+ side is shorter than size, then a center crop."""
50
+ w, h = img.size
51
+ if w < size or h < size:
52
+ s = max(size / w, size / h)
53
+ img = img.resize((max(int(w * s) + 1, size), max(int(h * s) + 1, size)), Image.BICUBIC)
54
+ w, h = img.size
55
+ x, y = (w - size) // 2, (h - size) // 2
56
+ return img.crop((x, y, x + size, y + size))
57
+
58
+
59
+ def main() -> None:
60
  ap = argparse.ArgumentParser()
61
+ ap.add_argument("--model", required=True, help="release folder, e.g. gblsr-scalar-asr")
62
+ ap.add_argument("--image", help="input image; a random input when omitted")
63
+ ap.add_argument("--scale", type=float, default=4.0, help="upscaling factor for super-resolution models")
64
+ ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
65
+ ap.add_argument("--out", help="where to write the output image")
66
  args = ap.parse_args()
67
 
 
68
  model, cfg = load_model(args.model)
69
+ model = model.to(args.device)
70
+ native = "build" in cfg
 
71
  if args.image:
72
+ img = Image.open(args.image).convert("RGB")
73
+ if native:
74
+ img = standardize(img, cfg["build"]["image_size"])
75
+ x = torch.from_numpy(np.asarray(img, dtype=np.float32) / 255.0).permute(2, 0, 1).unsqueeze(0)
 
76
  else:
77
+ size = cfg["build"]["image_size"] if native else 64
78
+ x = torch.rand(1, 3, size, size)
79
+ x = x.to(args.device)
80
  with torch.no_grad():
81
+ if native:
82
+ out = model(x)["recon"]
 
83
  else:
84
+ out = model.predict_full(x, H_q=round(x.shape[-2] * args.scale), W_q=round(x.shape[-1] * args.scale),
85
+ tile_q=30000)
86
  out = out.clamp(0, 1)
87
  print(f"{args.model}: input {tuple(x.shape)} -> output {tuple(out.shape)}")
 
88
  if args.out:
89
+ arr = (out[0].permute(1, 2, 0).cpu().numpy() * 255).round().astype(np.uint8)
90
+ Image.fromarray(arr).save(args.out)
 
 
 
91
  print(f"wrote {args.out}")
92
 
93
 
gblsr-scalar-p16/config.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "gblsr-scalar-p16",
3
+ "build": {
4
+ "arm": "local_spectral",
5
+ "image_size": 256,
6
+ "patch_size": 16,
7
+ "p_max": 16,
8
+ "d_feat": 128,
9
+ "s_e_range": [
10
+ 0.25,
11
+ 2.0
12
+ ],
13
+ "bandwidth_mode": "global_scalar",
14
+ "adapt_order": false
15
+ },
16
+ "seed": 0,
17
+ "trainable_params": 842115,
18
+ "trained_steps": 1000000,
19
+ "learned_bandwidth": 0.658
20
+ }
{gblsr-scalar → gblsr-scalar-p16}/model.safetensors RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:02ce1056cb8b0e7594d72c161de3fc30d0471bee07e516ddfbd974385d3f1e58
3
- size 3962604
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:40b6ed483353a731329ddce156cccbaee1e750c18b564280ae4b54b43e72b297
3
+ size 3370836
{gblsr-scalar → gblsr-scalar-p32}/config.json RENAMED
@@ -1,15 +1,20 @@
1
  {
2
- "model": "gblsr-scalar",
3
  "build": {
4
  "arm": "local_spectral",
 
5
  "patch_size": 32,
6
  "p_max": 16,
7
- "bandwidth_mode": "global_scalar",
8
- "adapt_order": false,
9
  "d_feat": 128,
10
- "image_size": 256
 
 
 
 
 
11
  },
12
  "seed": 0,
13
  "trainable_params": 989955,
14
- "trained_steps": 2000
 
15
  }
 
1
  {
2
+ "model": "gblsr-scalar-p32",
3
  "build": {
4
  "arm": "local_spectral",
5
+ "image_size": 256,
6
  "patch_size": 32,
7
  "p_max": 16,
 
 
8
  "d_feat": 128,
9
+ "s_e_range": [
10
+ 0.25,
11
+ 2.0
12
+ ],
13
+ "bandwidth_mode": "global_scalar",
14
+ "adapt_order": false
15
  },
16
  "seed": 0,
17
  "trainable_params": 989955,
18
+ "trained_steps": 1000000,
19
+ "learned_bandwidth": 0.7585
20
  }
gblsr-scalar-p32/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b2269b67cc0bb81b5d2111f2e7573b936ef914486a9fd9e5e20b758ff7776d6
3
+ size 3962548