Soo Celinta Transformer SwinIR

SwinIR, short for Swin Image Restoration, is a neural network that takes a damaged or low-resolution image and outputs a cleaner or larger version. Image restoration covers several problems: super-resolution, which enlarges an image while inventing plausible fine detail; denoising, which removes random grain; and JPEG compression artifact reduction, which removes the blocky patterns and color smearing that heavy compression leaves behind. SwinIR handles all of them with one architecture trained separately for each task. It was introduced by Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool and Radu Timofte of the Computer Vision Lab at ETH Zurich, with KU Leuven, in a paper posted to arXiv on August 23, 2021.

Before SwinIR, the best restoration models were convolutional neural networks (CNNs), which slide small learned filters across an image. The SwinIR paper points to two weaknesses of convolution for this job: the same filter is applied to every region regardless of content, and each filter sees only a small neighborhood, so long-range patterns are hard to use. Transformers, which use self-attention to let every position weigh information from other positions based on content, address both, but full self-attention over every pixel pair is far too expensive for large images. The earlier transformer model IPT also needed more than 115.5 million parameters and pretraining on ImageNet.

SwinIR's answer is the Swin Transformer, a vision backbone posted to arXiv in March 2021 by Ze Liu and colleagues. Swin computes attention only inside small non-overlapping windows, which keeps cost linear in image size rather than quadratic, and it alternates between a regular window grid and one shifted by half a window so that information flows across window borders from one layer to the next. SwinIR adapts this to restoration in three stages. A single 3 by 3 convolution extracts shallow features from the input. A deep feature extraction module then applies a stack of residual Swin Transformer blocks (RSTBs); each contains several Swin Transformer layers followed by a convolution and a skip connection that adds the block's input back to its output. Finally, a reconstruction module combines shallow and deep features and, for super-resolution, enlarges them with a sub-pixel convolution, a layer that rearranges extra feature channels into extra pixels.

The default configuration in the paper uses 6 RSTBs, 6 Swin layers per block, an 8 by 8 attention window, 180 feature channels and 6 attention heads, about 11.8 million parameters for classical super-resolution. A lightweight version cuts this to 4 blocks and 60 channels, under 900,000 parameters. Super-resolution models are trained with an L1 loss, the average absolute pixel difference from the ground truth; denoising and JPEG models use the closely related Charbonnier loss. For real-world photos, where no perfect ground truth exists, the model is retrained on images degraded with the BSRGAN degradation model and trained with a mix of pixel, adversarial (GAN) and perceptual losses to look sharper.

On benchmarks, the paper reports state-of-the-art results on all three task families. In classical super-resolution trained on the DIV2K dataset, SwinIR's largest gain over prior methods was 0.26 dB of peak signal-to-noise ratio (PSNR, a pixel-level fidelity score in decibels) on Manga109 at 4x, and up to 0.47 dB more when trained on DIV2K plus Flickr2K. It beat IPT while using 11.8 million parameters instead of 115.5 million. In lightweight super-resolution it gained up to 0.53 dB at a similar size to competitors. In JPEG artifact reduction it gained at least 0.11 dB and 0.07 dB on two test sets over DRUNet with 11.5 million parameters against DRUNet's 32.7 million. The abstract summarizes the gains as 0.14 to 0.45 dB with up to 67% fewer parameters.

These gains came with costs. The paper reports that on a 1,024 by 1,024 image, the CNN model RCAN took about 0.2 seconds, SwinIR about 1.1 seconds and IPT about 4.5 seconds. The SwinIR repository gives a closer comparison on Manga109 at 4x: SwinIR scored 31.67 dB against RCAN's 31.22 dB with fewer parameters (11.9 million against 15.6 million) and slightly fewer operations, but ran about three times slower on a 256 by 256 input (0.539 against 0.180 seconds) and used more memory. Attention over windows is less hardware-friendly than plain convolution, so fewer parameters does not mean faster inference.

SwinIR also shares the limits of all restoration models. Each model is trained for one task and one degradation level, such as 4x upscaling or noise level 25, and does poorly on inputs that do not match. PSNR rewards blurry averages, so the highest-scoring models can look soft, while GAN-trained real-world models look sharper but can invent textures that were never in the scene. That makes upscaled images unsuitable as evidence of fine detail, for example in forensic or medical settings. Later analysis in the HAT paper (2022) found that window-based super-resolution transformers use only a limited spatial range of the input, which motivated designs that widen it.

In practice SwinIR is easy to use. The official code and pretrained models are released under the Apache 2.0 license, and the repository reports training a classical model in about 1.8 days on eight Nvidia RTX 2080 Ti GPUs for 500,000 iterations. For most people the simplest route is a tool that bundles it, such as the AUTOMATIC1111 web UI's upscaler menu, which lists SwinIR and Swin2SR alongside Real-ESRGAN and ESRGAN.