Histoformer verified input/output samples across rain, raindrop, and snow
Task
Domain
Venue
Params
License
Easy-to-use mirror of Histoformer, the ECCV 2024 all-weather image restoration model (handles rain, raindrops, and snow in a single unified network) from Sun et al. This card exists to make the pretrained model simple to load and run in a few lines of Python — the original repository ships the full research codebase (BasicSR training framework, distributed-training configs, dataset generation scripts) behind CLI-only, multi-step instructions, which makes plain "just run inference" usage harder than it needs to be.
Disclaimer
This is not an official release. All credit for the method, the model, and the pretrained weights belongs entirely to the original authors: Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao. This repository claims no contribution to the underlying research, architecture, or training — it packages the same pretrained weights the authors already released, with clearer documentation and a minimal usage path.
Why this exists, concretely — not as criticism of the original work, just the gap this card fills:
The original repo's README documents usage as a multi-step CLI flow (cd Allweather, download weights into a specific folder structure, edit/point a YAML config, run test_histoformer.py) that assumes a full clone of the research codebase.
The original HF repo mirrors the entire project (training scripts, BasicSR framework internals, setup.py, etc.) rather than presenting itself as a loadable model — there's no minimal "load model, run image, get output" path documented.
We verified the actual minimal path ourselves (see Quickstart) — it turns out to be about 10 lines of plain PyTorch, no BasicSR framework or config files required for inference.
Please cite the original papers if you use this model (see Citation), and refer to the official repository for training code, or if you want the full research codebase.
What is Histoformer
Most transformer-based restoration methods reduce self-attention's cost by restricting it to the channel dimension or to small fixed spatial windows, which limits their ability to capture long-range spatial structure. Histoformer instead sorts and segments spatial features into intensity-based histogram bins, then applies self-attention across and within those bins — grouping similarly-degraded pixels together regardless of where they are in the image, rather than by spatial proximity. Since rain, raindrops, and snow all cause broadly similar occlusion/brightness patterns, this lets a single model handle all three degradation types without task-specific branches.
Params: 16,615,100 (verified by loading the checkpoint directly — see Quickstart)
Checkpoint size: ~64 MB
Architecture family: 4-level U-shaped Transformer encoder-decoder (same general shape as Restormer), with histogram self-attention replacing standard channel/window attention
Available Checkpoints
The original release ships two checkpoints, trained/fine-tuned differently — this distinction isn't clearly spelled out in the original README, so worth being explicit here:
Real-world images, e.g. the RealSnow benchmark or your own photos
If you're not sure which to use on a real photo (not a benchmark image), start with net_g_real.pth.
Quickstart
python
1import torch
2from huggingface_hub import hf_hub_download
3from PIL import Image
4import numpy as np
56# 1. Get the architecture definition (from the original repo — it's a single7# self-contained file with no BasicSR framework dependency for inference)8# git clone https://github.com/sunshangquan/Histoformer and add9# `Histoformer/basicsr` to your path, or copy `histoformer_arch.py` directly.10from basicsr.models.archs.histoformer_arch import Histoformer
1112# 2. Build the model from the published config (the exact hyperparameters13# used for training, from Allweather_Histoformer.yml)14import json
15config_path = hf_hub_download(repo_id="dronefreak/Histoformer", filename="config.json")16config = json.load(open(config_path))17config.pop("architecture")# not a constructor arg18model = Histoformer(**config)1920# 3. Download and load a checkpoint21weights_path = hf_hub_download(repo_id="dronefreak/Histoformer", filename="net_g_real.pth")22ckpt = torch.load(weights_path, map_location="cpu", weights_only=False)23model.load_state_dict(ckpt["params"])24model.eval()2526# 4. Run inference (pad to a multiple of 8 — the network downsamples 3x by /2)27img = Image.open("your_image.jpg").convert("RGB")28arr = np.array(img).astype(np.float32)/255.029t = torch.from_numpy(arr).permute(2,0,1).unsqueeze(0)3031_, _, h, w = t.shape
32pad_h, pad_w =(8- h %8)%8,(8- w %8)%833t_padded = torch.nn.functional.pad(t,(0, pad_w,0, pad_h), mode="reflect")3435with torch.no_grad():36 out = model(t_padded)[:,:,:h,:w].clamp(0,1)3738out_img = Image.fromarray((out[0].permute(1,2,0).numpy()*255).astype(np.uint8))39out_img.save("restored.jpg")
Runs on CPU (~20s for a 720×480 image on a modern desktop CPU, verified) or GPU (much faster). No BasicSR training framework, no YAML config parsing, no distributed-training setup needed for inference — just the architecture file and the checkpoint.
Evaluation (as reported in the original paper)
These are the authors' own reported numbers (arXiv:2407.10172, Table 1) — we have not independently reproduced them; we've only verified the model loads correctly and produces visually sensible output (see the banner above and Disclaimer). Take these as the original paper's claims, not this card's independent measurement.
The paper reports these as state-of-the-art among unified all-weather methods at publication time (outperforming TransWeather, WGWSNet, WeatherDiff).
Benchmark sources
Li, Cheong & Tan, Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning, CVPR 2019, arXiv:1904.05050 (Outdoor-Rain / Test1).
Qian et al., Attentive Generative Adversarial Network for Raindrop Removal from a Single Image, CVPR 2018, arXiv:1711.10098 (RainDrop).
Liu et al., DesnowNet: Context-Aware Deep Network for Snow Removal, IEEE TIP 2018, arXiv:1708.04512 (Snow100K-S/L).
ClearView Cross-Domain Check
Separately, ClearView ran this checkpoint (net_g_real.pth) through its own evaluation pipeline across 10 rain/rain+fog test sets, not the benchmarks above. This is not a reproduction of the paper's numbers (different benchmarks, different pipeline), just PSNR/SSIM on a separate set of test sets for cross-domain context.
Yang et al., Deep Joint Rain Detection and Removal from a Single Image, CVPR 2017, arXiv:1609.07769 (Rain100H/L).
Zhang & Patel, Density-aware Single Image De-raining using a Multi-stream Dense Network, CVPR 2018, arXiv:1802.07412 (Test100).
Zhang, Sindagi & Patel, Image De-raining Using a Conditional Generative Adversarial Network, IEEE TCSVT 2019, arXiv:1701.05957 (Test1200).
Fu et al., Removing Rain from Single Images via a Deep Detail Network, CVPR 2017, CVF open access (Test2800 / DDN-Data / Rain1400). No dedicated arXiv preprint found for this one, only the CVPR proceedings version (not to be confused with the same authors' related but distinct arXiv:1609.02087, "Clearing the Skies").
Wang et al., Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset, CVPR 2019, arXiv:1904.01538 (SPA-Data).
Li et al., Toward Real-world Single Image Deraining: A New Benchmark and Beyond, arXiv:2206.05514, 2022 (RealRain-1k-H/L).
Li, Cheong & Tan, Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning, CVPR 2019, arXiv:1904.05050 (AllWeather rain+fog / Outdoor-Rain).
Training Data
Histoformer is trained on a composite of independently-published datasets — the same benchmarks are used for testing:
Outdoor-Rain — Li et al., Heavy Rain Image Restoration, CVPR 2019
Snow100K — Liu et al., DesnowNet, TIP 2018 (arXiv:1708.04512)
RainDrop — Qian et al., Attentive GAN for Raindrop Removal, CVPR 2018
This model card does not redistribute the training or test data — only the pretrained weights. The standard test benchmarks (Outdoor-Rain/Test1, Snow100K-S/L, RainDrop) are readily available as a single bundle from the original authors: Google Drive.
License
The original HF release (sunsean/Histoformer) states MIT in its model card metadata — unlike the GitHub repository, which has no LICENSE file. This mirror is distributed under the same MIT terms.
Citation
If you use this model, please cite the original work:
bibtex
1@article{sun2024restoring,
2 title={Restoring Images in Adverse Weather Conditions via Histogram Transformer},
3 author={Sun, Shangquan and Ren, Wenqi and Gao, Xinwei and Wang, Rui and Cao, Xiaochun},
4 journal={arXiv preprint arXiv:2407.10172},
5 year={2024}
6}
78@InProceedings{10.1007/978-3-031-72670-5_7,
9 author="Sun, Shangquan and Ren, Wenqi and Gao, Xinwei and Wang, Rui and Cao, Xiaochun",
10 title="Restoring Images in Adverse Weather Conditions via Histogram Transformer",
11 booktitle="Computer Vision -- ECCV 2024",
12 year="2025",
13 publisher="Springer Nature Switzerland",
14 pages="111--129",
15 isbn="978-3-031-72670-5"
16}
Acknowledgements
We sincerely thank Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao for developing Histoformer and publicly releasing the pretrained weights under a permissive license.
