README.md

---
language:
- en

license: mit

pipeline_tag: image-to-image

tags:
- image-restoration
- adverse-weather
- deraining
- desnowing
- dehazing
- raindrop-removal
- computer-vision
- pytorch
- eccv2024
---

# Histoformer: All-Weather Image Restoration

<p align="center">
  <img src="banner.jpg" alt="Histoformer verified input/output samples across rain, raindrop, and snow"/>
</p>

![Task](https://img.shields.io/badge/Task-Image%20Restoration-blue?style=flat-square)
![Domain](https://img.shields.io/badge/Domain-All--Weather-0aa1a7?style=flat-square)
![Venue](https://img.shields.io/badge/ECCV-2024-orange?style=flat-square)
![Params](https://img.shields.io/badge/Params-16.6M-success?style=flat-square)
![License](https://img.shields.io/badge/License-MIT-lightgrey?style=flat-square)

> **Easy-to-use mirror of Histoformer**, the ECCV 2024 all-weather image restoration model (handles rain, raindrops, and snow in a single unified network) from Sun et al. This card exists to make the pretrained model simple to load and run in a few lines of Python — the original repository ships the full research codebase (BasicSR training framework, distributed-training configs, dataset generation scripts) behind CLI-only, multi-step instructions, which makes plain "just run inference" usage harder than it needs to be.

## Disclaimer

This is **not** an official release. All credit for the method, the model, and the pretrained weights belongs entirely to the original authors: **Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao**. This repository claims no contribution to the underlying research, architecture, or training — it packages the same pretrained weights the authors already released, with clearer documentation and a minimal usage path.

**Why this exists**, concretely — not as criticism of the original work, just the gap this card fills:
- The original repo's README documents usage as a multi-step CLI flow (`cd Allweather`, download weights into a specific folder structure, edit/point a YAML config, run `test_histoformer.py`) that assumes a full clone of the research codebase.
- The original HF repo mirrors the *entire* project (training scripts, BasicSR framework internals, `setup.py`, etc.) rather than presenting itself as a loadable model — there's no minimal "load model, run image, get output" path documented.
- We verified the actual minimal path ourselves (see [Quickstart](#quickstart)) — it turns out to be about 10 lines of plain PyTorch, no BasicSR framework or config files required for inference.

Please cite the original papers if you use this model (see [Citation](#citation)), and refer to the [official repository](https://github.com/sunshangquan/Histoformer) for training code, or if you want the full research codebase.

---

# What is Histoformer

Most transformer-based restoration methods reduce self-attention's cost by restricting it to the channel dimension or to small fixed spatial windows, which limits their ability to capture long-range spatial structure. Histoformer instead sorts and segments spatial features into **intensity-based histogram bins**, then applies self-attention across and within those bins — grouping similarly-degraded pixels together regardless of where they are in the image, rather than by spatial proximity. Since rain, raindrops, and snow all cause broadly similar occlusion/brightness patterns, this lets a single model handle all three degradation types without task-specific branches.

- **Paper**: [Restoring Images in Adverse Weather Conditions via Histogram Transformer](https://arxiv.org/abs/2407.10172), ECCV 2024
- **Params**: 16,615,100 (verified by loading the checkpoint directly — see [Quickstart](#quickstart))
- **Checkpoint size**: ~64 MB
- **Architecture family**: 4-level U-shaped Transformer encoder-decoder (same general shape as Restormer), with histogram self-attention replacing standard channel/window attention

---

# Available Checkpoints

The original release ships **two** checkpoints, trained/fine-tuned differently — this distinction isn't clearly spelled out in the original README, so worth being explicit here:

| Checkpoint | Trained on | Best for |
|---|---|---|
| `net_g_best.pth` | Synthetic all-weather composite (Outdoor-Rain + Snow100K + RainDrop) | Synthetic-style benchmarks: Test1, Snow100K-S/L, RainDrop |
| `net_g_real.pth` | Fine-tuned toward real-world photos | Real-world images, e.g. the RealSnow benchmark or your own photos |

If you're not sure which to use on a real photo (not a benchmark image), start with `net_g_real.pth`.

---

# Quickstart

```python
import torch
from huggingface_hub import hf_hub_download
from PIL import Image
import numpy as np

# 1. Get the architecture definition (from the original repo — it's a single
#    self-contained file with no BasicSR framework dependency for inference)
#    git clone https://github.com/sunshangquan/Histoformer and add
#    `Histoformer/basicsr` to your path, or copy `histoformer_arch.py` directly.
from basicsr.models.archs.histoformer_arch import Histoformer

# 2. Build the model from the published config (the exact hyperparameters
#    used for training, from Allweather_Histoformer.yml)
import json
config_path = hf_hub_download(repo_id="dronefreak/Histoformer", filename="config.json")
config = json.load(open(config_path))
config.pop("architecture")  # not a constructor arg
model = Histoformer(**config)

# 3. Download and load a checkpoint
weights_path = hf_hub_download(repo_id="dronefreak/Histoformer", filename="net_g_real.pth")
ckpt = torch.load(weights_path, map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["params"])
model.eval()

# 4. Run inference (pad to a multiple of 8 — the network downsamples 3x by /2)
img = Image.open("your_image.jpg").convert("RGB")
arr = np.array(img).astype(np.float32) / 255.0
t = torch.from_numpy(arr).permute(2, 0, 1).unsqueeze(0)

_, _, h, w = t.shape
pad_h, pad_w = (8 - h % 8) % 8, (8 - w % 8) % 8
t_padded = torch.nn.functional.pad(t, (0, pad_w, 0, pad_h), mode="reflect")

with torch.no_grad():
    out = model(t_padded)[:, :, :h, :w].clamp(0, 1)

out_img = Image.fromarray((out[0].permute(1, 2, 0).numpy() * 255).astype(np.uint8))
out_img.save("restored.jpg")
```

Runs on CPU (~20s for a 720×480 image on a modern desktop CPU, verified) or GPU (much faster). No BasicSR training framework, no YAML config parsing, no distributed-training setup needed for inference — just the architecture file and the checkpoint.

---

# Evaluation (as reported in the original paper)

These are the authors' own reported numbers (arXiv:2407.10172, Table 1) — we have **not** independently reproduced them; we've only verified the model loads correctly and produces visually sensible output (see the banner above and [Disclaimer](#disclaimer)). Take these as the original paper's claims, not this card's independent measurement.

| Benchmark | Degradation | PSNR | SSIM |
|---|---|---|---|
| Outdoor-Rain (Test1) [[1]](#benchmark-sources) | Rain + fog | 32.08 | 0.9389 |
| RainDrop [[2]](#benchmark-sources) | Adherent raindrops | 33.06 | 0.9441 |
| Snow100K-S [[3]](#benchmark-sources) | Light snow | 37.41 | 0.9656 |
| Snow100K-L [[3]](#benchmark-sources) | Heavy snow | 32.16 | 0.9261 |

The paper reports these as state-of-the-art among unified all-weather methods at publication time (outperforming TransWeather, WGWSNet, WeatherDiff).

#### Benchmark sources

1. Li, Cheong & Tan, *Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning*, CVPR 2019, [arXiv:1904.05050](https://arxiv.org/abs/1904.05050) (Outdoor-Rain / Test1).
2. Qian et al., *Attentive Generative Adversarial Network for Raindrop Removal from a Single Image*, CVPR 2018, [arXiv:1711.10098](https://arxiv.org/abs/1711.10098) (RainDrop).
3. Liu et al., *DesnowNet: Context-Aware Deep Network for Snow Removal*, IEEE TIP 2018, [arXiv:1708.04512](https://arxiv.org/abs/1708.04512) (Snow100K-S/L).

### ClearView Cross-Domain Check

Separately, [ClearView](https://github.com/dronefreak/clearview) ran this checkpoint (`net_g_real.pth`) through its own evaluation pipeline across 10 rain/rain+fog test sets, not the benchmarks above. This is **not** a reproduction of the paper's numbers (different benchmarks, different pipeline), just PSNR/SSIM on a separate set of test sets for cross-domain context.

| Test Set | Domain | PSNR | SSIM |
|---|---|---|---|
| Rain100L [[1]](#test-set-sources) | Synthetic | 25.83 | 0.836 |
| Rain100H [[1]](#test-set-sources) | Synthetic | 12.22 | 0.364 |
| Test100 [[2]](#test-set-sources) | Synthetic | 22.01 | 0.684 |
| Test1200 [[3]](#test-set-sources) | Synthetic | 24.20 | 0.727 |
| Test2800 [[4]](#test-set-sources) | Synthetic | 24.71 | 0.785 |
| DDN-Data [[4]](#test-set-sources) | Synthetic | 25.04 | 0.784 |
| SPA-Data [[5]](#test-set-sources) | Real-world | 32.18 | 0.929 |
| RealRain-1k-H [[6]](#test-set-sources) | Real-world | 21.86 | 0.761 |
| RealRain-1k-L [[6]](#test-set-sources) | Real-world | 25.47 | 0.867 |
| AllWeather (rain+fog) [[7]](#test-set-sources) | Cross-domain (stress) | 30.75 | 0.923 |

#### Test set sources

1. Yang et al., *Deep Joint Rain Detection and Removal from a Single Image*, CVPR 2017, [arXiv:1609.07769](https://arxiv.org/abs/1609.07769) (Rain100H/L).
2. Zhang & Patel, *Density-aware Single Image De-raining using a Multi-stream Dense Network*, CVPR 2018, [arXiv:1802.07412](https://arxiv.org/abs/1802.07412) (Test100).
3. Zhang, Sindagi & Patel, *Image De-raining Using a Conditional Generative Adversarial Network*, IEEE TCSVT 2019, [arXiv:1701.05957](https://arxiv.org/abs/1701.05957) (Test1200).
4. Fu et al., *Removing Rain from Single Images via a Deep Detail Network*, CVPR 2017, [CVF open access](https://openaccess.thecvf.com/content_cvpr_2017/papers/Fu_Removing_Rain_From_CVPR_2017_paper.pdf) (Test2800 / DDN-Data / Rain1400). No dedicated arXiv preprint found for this one, only the CVPR proceedings version (not to be confused with the same authors' related but distinct arXiv:1609.02087, "Clearing the Skies").
5. Wang et al., *Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset*, CVPR 2019, [arXiv:1904.01538](https://arxiv.org/abs/1904.01538) (SPA-Data).
6. Li et al., *Toward Real-world Single Image Deraining: A New Benchmark and Beyond*, [arXiv:2206.05514](https://arxiv.org/abs/2206.05514), 2022 (RealRain-1k-H/L).
7. Li, Cheong & Tan, *Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning*, CVPR 2019, [arXiv:1904.05050](https://arxiv.org/abs/1904.05050) (AllWeather rain+fog / Outdoor-Rain).

---

# Training Data

Histoformer is trained on a composite of independently-published datasets — the same benchmarks are used for testing:

- **Outdoor-Rain** — Li et al., *Heavy Rain Image Restoration*, CVPR 2019
- **Snow100K** — Liu et al., *DesnowNet*, TIP 2018 (arXiv:1708.04512)
- **RainDrop** — Qian et al., *Attentive GAN for Raindrop Removal*, CVPR 2018

This model card does not redistribute the training or test data — only the pretrained weights. The standard test benchmarks (Outdoor-Rain/Test1, Snow100K-S/L, RainDrop) are readily available as a single bundle from the original authors: [Google Drive](https://drive.google.com/file/d/1tfeBnjZX1wIhIFPl6HOzzOKOyo0GdGHl/view).

---

# License

The original HF release ([`sunsean/Histoformer`](https://huggingface.co/sunsean/Histoformer)) states **MIT** in its model card metadata — unlike the GitHub repository, which has no LICENSE file. This mirror is distributed under the same MIT terms.

---

# Citation

If you use this model, please cite the original work:

```bibtex
@article{sun2024restoring,
  title={Restoring Images in Adverse Weather Conditions via Histogram Transformer},
  author={Sun, Shangquan and Ren, Wenqi and Gao, Xinwei and Wang, Rui and Cao, Xiaochun},
  journal={arXiv preprint arXiv:2407.10172},
  year={2024}
}

@InProceedings{10.1007/978-3-031-72670-5_7,
    author="Sun, Shangquan and Ren, Wenqi and Gao, Xinwei and Wang, Rui and Cao, Xiaochun",
    title="Restoring Images in Adverse Weather Conditions via Histogram Transformer",
    booktitle="Computer Vision -- ECCV 2024",
    year="2025",
    publisher="Springer Nature Switzerland",
    pages="111--129",
    isbn="978-3-031-72670-5"
}
```

---

# Acknowledgements

We sincerely thank Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao for developing Histoformer and publicly releasing the pretrained weights under a permissive license.