Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
dronefreak 
posted an update about 12 hours ago
Post
62
🌧️❄️ Free demo: remove rain, raindrops, or snow from a photo with a single model

I put together an unofficial demo for **Histoformer** (ECCV 2024, arXiv: 2407.10172), a 16.6M-parameter transformer that handles three different weather degradations, rain streaks, adherent raindrops, and snow, in one unified model. It uses a "histogram self-attention" mechanism that groups pixels by degradation intensity instead of spatial position, which is a
neat way to sidestep the usual spatial-window tradeoffs in restoration transformers.

Try it here, free on ZeroGPU: dronefreak/histoformer-weather-restoration

Upload a photo and get a before/after slider. Two checkpoints available: one tuned for real-world photos, one for the paper's synthetic benchmarks.

Also put together a cleaner, easy-to-use model card with a copy-pasteable Quickstart if you'd rather run it yourself: dronefreak/Histoformer

This is an unofficial demo/mirror, not affiliated with the original authors. All credit for the actual research goes to Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao (@sunsean ). Official repo: https://github.com/sunshangquan/Histoformer. Weights are MIT-licensed.

Reported numbers from the paper: 32.1 PSNR on rain+fog (Outdoor-Rain), 33.1 on raindrops, 37.4 / 32.2 on light/heavy snow (Snow100K-S/L).

The histogram self-attention angle is a useful way to frame restoration: grouping pixels by degradation intensity may reduce the cost of purely spatial windows. The two-checkpoint comparison is practical because synthetic gains can hide real-photo failures. Where does the real-world checkpoint diverge most from the Snow100K or rain-plus-fog results?

·

The honest answer is that neither the paper nor this card has a direct number for that. The 32.08 / 33.06 / 37.41 / 32.16 PSNR figures (Outdoor-Rain, RainDrop, Snow100K-S, Snow100K-L) are all for net_g_best, the checkpoint trained on the synthetic composite, evaluated against those same synthetic test sets. The paper's real-world comparison (RealSnow, 1,329 actual photos from Snow100K's real-world split) is qualitative only, Figure 6 in the paper shows side-by-side crops, there's no PSNR/SSIM table for it. That's not an oversight on the authors' part either, RealSnow has no paired clean ground truth, so a full-reference metric like PSNR isn't computable for it the same way it is for the synthetic sets. That absence of a number is itself informative: it means "where does it diverge" can't currently be answered with a metric, only by looking.

If I reason about it qualitatively rather than measuring it: the usual gap for this kind of model shows up where the synthetic renderer's assumptions break down. Snow100K's synthetic particles are fairly uniform in size and motion; real snow varies a lot more in particle size, translucency, and motion blur within a single frame, so I'd expect net_g_real to hold up better there than net_g_best would on the same photo. Outdoor-Rain's fog is a synthetic atmospheric-scattering model applied on top of rain streaks; real rain-plus-fog scenes usually also mix in uneven lighting and haze density that the synthetic pipeline doesn't fully capture, so that's probably where the bigger real-world gap sits too.

That's domain reasoning, not a measured result, and I'd rather say that plainly than imply a number I don't have. If it's useful, I do have both checkpoints and the real vs synthetic test sets locally already (that's what the demo's checkpoint selector is built from), so I could run an actual informal side-by-side and post real examples if there's interest. Happy to do that as a follow-up if it'd help more than the qualitative answer above.