--- language: - en license: mit pipeline_tag: image-to-image tags: - image-restoration - adverse-weather - deraining - desnowing - dehazing - raindrop-removal - computer-vision - pytorch - eccv2024 --- # Histoformer: All-Weather Image Restoration

Histoformer verified input/output samples across rain, raindrop, and snow

![Task](https://img.shields.io/badge/Task-Image%20Restoration-blue?style=flat-square) ![Domain](https://img.shields.io/badge/Domain-All--Weather-0aa1a7?style=flat-square) ![Venue](https://img.shields.io/badge/ECCV-2024-orange?style=flat-square) ![Params](https://img.shields.io/badge/Params-16.6M-success?style=flat-square) ![License](https://img.shields.io/badge/License-MIT-lightgrey?style=flat-square) > **Easy-to-use mirror of Histoformer**, the ECCV 2024 all-weather image restoration model (handles rain, raindrops, and snow in a single unified network) from Sun et al. This card exists to make the pretrained model simple to load and run in a few lines of Python — the original repository ships the full research codebase (BasicSR training framework, distributed-training configs, dataset generation scripts) behind CLI-only, multi-step instructions, which makes plain "just run inference" usage harder than it needs to be. ## Disclaimer This is **not** an official release. All credit for the method, the model, and the pretrained weights belongs entirely to the original authors: **Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao**. This repository claims no contribution to the underlying research, architecture, or training — it packages the same pretrained weights the authors already released, with clearer documentation and a minimal usage path. **Why this exists**, concretely — not as criticism of the original work, just the gap this card fills: - The original repo's README documents usage as a multi-step CLI flow (`cd Allweather`, download weights into a specific folder structure, edit/point a YAML config, run `test_histoformer.py`) that assumes a full clone of the research codebase. - The original HF repo mirrors the *entire* project (training scripts, BasicSR framework internals, `setup.py`, etc.) rather than presenting itself as a loadable model — there's no minimal "load model, run image, get output" path documented. - We verified the actual minimal path ourselves (see [Quickstart](#quickstart)) — it turns out to be about 10 lines of plain PyTorch, no BasicSR framework or config files required for inference. Please cite the original papers if you use this model (see [Citation](#citation)), and refer to the [official repository](https://github.com/sunshangquan/Histoformer) for training code, or if you want the full research codebase. --- # What is Histoformer Most transformer-based restoration methods reduce self-attention's cost by restricting it to the channel dimension or to small fixed spatial windows, which limits their ability to capture long-range spatial structure. Histoformer instead sorts and segments spatial features into **intensity-based histogram bins**, then applies self-attention across and within those bins — grouping similarly-degraded pixels together regardless of where they are in the image, rather than by spatial proximity. Since rain, raindrops, and snow all cause broadly similar occlusion/brightness patterns, this lets a single model handle all three degradation types without task-specific branches. - **Paper**: [Restoring Images in Adverse Weather Conditions via Histogram Transformer](https://arxiv.org/abs/2407.10172), ECCV 2024 - **Params**: 16,615,100 (verified by loading the checkpoint directly — see [Quickstart](#quickstart)) - **Checkpoint size**: ~64 MB - **Architecture family**: 4-level U-shaped Transformer encoder-decoder (same general shape as Restormer), with histogram self-attention replacing standard channel/window attention --- # Available Checkpoints The original release ships **two** checkpoints, trained/fine-tuned differently — this distinction isn't clearly spelled out in the original README, so worth being explicit here: | Checkpoint | Trained on | Best for | |---|---|---| | `net_g_best.pth` | Synthetic all-weather composite (Outdoor-Rain + Snow100K + RainDrop) | Synthetic-style benchmarks: Test1, Snow100K-S/L, RainDrop | | `net_g_real.pth` | Fine-tuned toward real-world photos | Real-world images, e.g. the RealSnow benchmark or your own photos | If you're not sure which to use on a real photo (not a benchmark image), start with `net_g_real.pth`. --- # Quickstart ```python import torch from huggingface_hub import hf_hub_download from PIL import Image import numpy as np # 1. Get the architecture definition (from the original repo — it's a single # self-contained file with no BasicSR framework dependency for inference) # git clone https://github.com/sunshangquan/Histoformer and add # `Histoformer/basicsr` to your path, or copy `histoformer_arch.py` directly. from basicsr.models.archs.histoformer_arch import Histoformer # 2. Build the model from the published config (the exact hyperparameters # used for training, from Allweather_Histoformer.yml) import json config_path = hf_hub_download(repo_id="dronefreak/Histoformer", filename="config.json") config = json.load(open(config_path)) config.pop("architecture") # not a constructor arg model = Histoformer(**config) # 3. Download and load a checkpoint weights_path = hf_hub_download(repo_id="dronefreak/Histoformer", filename="net_g_real.pth") ckpt = torch.load(weights_path, map_location="cpu", weights_only=False) model.load_state_dict(ckpt["params"]) model.eval() # 4. Run inference (pad to a multiple of 8 — the network downsamples 3x by /2) img = Image.open("your_image.jpg").convert("RGB") arr = np.array(img).astype(np.float32) / 255.0 t = torch.from_numpy(arr).permute(2, 0, 1).unsqueeze(0) _, _, h, w = t.shape pad_h, pad_w = (8 - h % 8) % 8, (8 - w % 8) % 8 t_padded = torch.nn.functional.pad(t, (0, pad_w, 0, pad_h), mode="reflect") with torch.no_grad(): out = model(t_padded)[:, :, :h, :w].clamp(0, 1) out_img = Image.fromarray((out[0].permute(1, 2, 0).numpy() * 255).astype(np.uint8)) out_img.save("restored.jpg") ``` Runs on CPU (~20s for a 720×480 image on a modern desktop CPU, verified) or GPU (much faster). No BasicSR training framework, no YAML config parsing, no distributed-training setup needed for inference — just the architecture file and the checkpoint. --- # Evaluation (as reported in the original paper) These are the authors' own reported numbers (arXiv:2407.10172, Table 1) — we have **not** independently reproduced them; we've only verified the model loads correctly and produces visually sensible output (see the banner above and [Disclaimer](#disclaimer)). Take these as the original paper's claims, not this card's independent measurement. | Benchmark | Degradation | PSNR | SSIM | |---|---|---|---| | Outdoor-Rain (Test1) [[1]](#benchmark-sources) | Rain + fog | 32.08 | 0.9389 | | RainDrop [[2]](#benchmark-sources) | Adherent raindrops | 33.06 | 0.9441 | | Snow100K-S [[3]](#benchmark-sources) | Light snow | 37.41 | 0.9656 | | Snow100K-L [[3]](#benchmark-sources) | Heavy snow | 32.16 | 0.9261 | The paper reports these as state-of-the-art among unified all-weather methods at publication time (outperforming TransWeather, WGWSNet, WeatherDiff). #### Benchmark sources 1. Li, Cheong & Tan, *Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning*, CVPR 2019, [arXiv:1904.05050](https://arxiv.org/abs/1904.05050) (Outdoor-Rain / Test1). 2. Qian et al., *Attentive Generative Adversarial Network for Raindrop Removal from a Single Image*, CVPR 2018, [arXiv:1711.10098](https://arxiv.org/abs/1711.10098) (RainDrop). 3. Liu et al., *DesnowNet: Context-Aware Deep Network for Snow Removal*, IEEE TIP 2018, [arXiv:1708.04512](https://arxiv.org/abs/1708.04512) (Snow100K-S/L). ### ClearView Cross-Domain Check Separately, [ClearView](https://github.com/dronefreak/clearview) ran this checkpoint (`net_g_real.pth`) through its own evaluation pipeline across 10 rain/rain+fog test sets, not the benchmarks above. This is **not** a reproduction of the paper's numbers (different benchmarks, different pipeline), just PSNR/SSIM on a separate set of test sets for cross-domain context. | Test Set | Domain | PSNR | SSIM | |---|---|---|---| | Rain100L [[1]](#test-set-sources) | Synthetic | 25.83 | 0.836 | | Rain100H [[1]](#test-set-sources) | Synthetic | 12.22 | 0.364 | | Test100 [[2]](#test-set-sources) | Synthetic | 22.01 | 0.684 | | Test1200 [[3]](#test-set-sources) | Synthetic | 24.20 | 0.727 | | Test2800 [[4]](#test-set-sources) | Synthetic | 24.71 | 0.785 | | DDN-Data [[4]](#test-set-sources) | Synthetic | 25.04 | 0.784 | | SPA-Data [[5]](#test-set-sources) | Real-world | 32.18 | 0.929 | | RealRain-1k-H [[6]](#test-set-sources) | Real-world | 21.86 | 0.761 | | RealRain-1k-L [[6]](#test-set-sources) | Real-world | 25.47 | 0.867 | | AllWeather (rain+fog) [[7]](#test-set-sources) | Cross-domain (stress) | 30.75 | 0.923 | #### Test set sources 1. Yang et al., *Deep Joint Rain Detection and Removal from a Single Image*, CVPR 2017, [arXiv:1609.07769](https://arxiv.org/abs/1609.07769) (Rain100H/L). 2. Zhang & Patel, *Density-aware Single Image De-raining using a Multi-stream Dense Network*, CVPR 2018, [arXiv:1802.07412](https://arxiv.org/abs/1802.07412) (Test100). 3. Zhang, Sindagi & Patel, *Image De-raining Using a Conditional Generative Adversarial Network*, IEEE TCSVT 2019, [arXiv:1701.05957](https://arxiv.org/abs/1701.05957) (Test1200). 4. Fu et al., *Removing Rain from Single Images via a Deep Detail Network*, CVPR 2017, [CVF open access](https://openaccess.thecvf.com/content_cvpr_2017/papers/Fu_Removing_Rain_From_CVPR_2017_paper.pdf) (Test2800 / DDN-Data / Rain1400). No dedicated arXiv preprint found for this one, only the CVPR proceedings version (not to be confused with the same authors' related but distinct arXiv:1609.02087, "Clearing the Skies"). 5. Wang et al., *Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset*, CVPR 2019, [arXiv:1904.01538](https://arxiv.org/abs/1904.01538) (SPA-Data). 6. Li et al., *Toward Real-world Single Image Deraining: A New Benchmark and Beyond*, [arXiv:2206.05514](https://arxiv.org/abs/2206.05514), 2022 (RealRain-1k-H/L). 7. Li, Cheong & Tan, *Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning*, CVPR 2019, [arXiv:1904.05050](https://arxiv.org/abs/1904.05050) (AllWeather rain+fog / Outdoor-Rain). --- # Training Data Histoformer is trained on a composite of independently-published datasets — the same benchmarks are used for testing: - **Outdoor-Rain** — Li et al., *Heavy Rain Image Restoration*, CVPR 2019 - **Snow100K** — Liu et al., *DesnowNet*, TIP 2018 (arXiv:1708.04512) - **RainDrop** — Qian et al., *Attentive GAN for Raindrop Removal*, CVPR 2018 This model card does not redistribute the training or test data — only the pretrained weights. The standard test benchmarks (Outdoor-Rain/Test1, Snow100K-S/L, RainDrop) are readily available as a single bundle from the original authors: [Google Drive](https://drive.google.com/file/d/1tfeBnjZX1wIhIFPl6HOzzOKOyo0GdGHl/view). --- # License The original HF release ([`sunsean/Histoformer`](https://huggingface.co/sunsean/Histoformer)) states **MIT** in its model card metadata — unlike the GitHub repository, which has no LICENSE file. This mirror is distributed under the same MIT terms. --- # Citation If you use this model, please cite the original work: ```bibtex @article{sun2024restoring, title={Restoring Images in Adverse Weather Conditions via Histogram Transformer}, author={Sun, Shangquan and Ren, Wenqi and Gao, Xinwei and Wang, Rui and Cao, Xiaochun}, journal={arXiv preprint arXiv:2407.10172}, year={2024} } @InProceedings{10.1007/978-3-031-72670-5_7, author="Sun, Shangquan and Ren, Wenqi and Gao, Xinwei and Wang, Rui and Cao, Xiaochun", title="Restoring Images in Adverse Weather Conditions via Histogram Transformer", booktitle="Computer Vision -- ECCV 2024", year="2025", publisher="Springer Nature Switzerland", pages="111--129", isbn="978-3-031-72670-5" } ``` --- # Acknowledgements We sincerely thank Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao for developing Histoformer and publicly releasing the pretrained weights under a permissive license.