The goal of image super resolution is to restore a high resolution (HR) image from a single low resolution (LR) image. The image below shows the ground truth (HR), the bicubic upscaling and model upscaling.
Comparing Bicubic upscaling against the models x4 upscaling on Set5 Image 4
Model description
The CARN model proposes an architecture that implements a cascading mechanism upon a residual network for accurate and lightweight image super-resolution.
This model also applies the balanced attention (BAM) method invented by Wang et al. (2021) to further improve the results.
Intended uses & limitations
You can use the pre-trained models for upscaling your images 2x, 3x and 4x. You can also use the trainer to train a model on your own dataset.
How to use
The model can be used with the super_image library:
pip install super-image
Here is how to use a pre-trained model to upscale your image:
python
1from super_image import CarnModel, ImageLoader
2from PIL import Image
3import requests
45url ='https://paperswithcode.com/media/datasets/Set5-0000002728-07a9793f_zA3bDjj.jpg'6image = Image.open(requests.get(url, stream=True).raw)78model = CarnModel.from_pretrained('eugenesiow/carn-bam', scale=2)# scale 2, 3 and 4 models available9inputs = ImageLoader.load_image(image)10preds = model(inputs)1112ImageLoader.save_image(preds,'./scaled_2x.png')# save the output 2x scaled image to `./scaled_2x.png`13ImageLoader.save_compare(inputs, preds,'./scaled_2x_compare.png')# save an output comparing the super-image with a bicubic scaling
The models for 2x, 3x and 4x image super resolution were pretrained on DIV2K, a dataset of 800 high-quality (2K resolution) images for training, augmented to 4000 images and uses a dev set of 100 validation images (images numbered 801 to 900).
Training procedure
Preprocessing
We follow the pre-processing and training method of Wang et al..
Low Resolution (LR) images are created by using bicubic interpolation as the resizing method to reduce the size of the High Resolution (HR) images by x2, x3 and x4 times.
During training, RGB patches with size of 64×64 from the LR input are used together with their corresponding HR patches.
Data augmentation is applied to the training set in the pre-processing stage where five images are created from the four corners and center of the original image.
We need the huggingface datasets library to download the data:
pip install datasets
The following code gets the data and preprocesses/augments the data.
python
1from datasets import load_dataset
2from super_image.data import EvalDataset, TrainDataset, augment_five_crop
34augmented_dataset = load_dataset('eugenesiow/Div2k','bicubic_x4', split='train')\
5.map(augment_five_crop, batched=True, desc="Augmenting Dataset")# download and augment the data with the five_crop method6train_dataset = TrainDataset(augmented_dataset)# prepare the train dataset for loading PyTorch DataLoader7eval_dataset = EvalDataset(load_dataset('eugenesiow/Div2k','bicubic_x4', split='validation'))# prepare the eval dataset for the PyTorch DataLoader
Pretraining
The model was trained on GPU. The training code is provided below:
python
1from super_image import Trainer, TrainingArguments, CarnModel, CarnConfig
23training_args = TrainingArguments(4 output_dir='./results',# output directory5 num_train_epochs=1000,# total number of training epochs6)78config = CarnConfig(9 scale=4,# train a model to upscale 4x10 bam=True,# apply balanced attention to the network11)12model = CarnModel(config)1314trainer = Trainer(15 model=model,# the instantiated model to be trained16 args=training_args,# training arguments, defined above17 train_dataset=train_dataset,# training dataset18 eval_dataset=eval_dataset # evaluation dataset19)2021trainer.train()
1@misc{wang2021bam,
2 title={BAM: A Lightweight and Efficient Balanced Attention Mechanism for Single Image Super Resolution},
3 author={Fanyi Wang and Haotian Hu and Cheng Shen},
4 year={2021},
5 eprint={2104.07566},
6 archivePrefix={arXiv},
7 primaryClass={eess.IV}
8}
bibtex
1@article{ahn2018fast,
2 title={Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network},
3 author={Ahn, Namhyuk and Kang, Byungkon and Sohn, Kyung-Ah},
4 journal={arXiv preprint arXiv:1803.08664},
5 year={2018}
6}
