ESRGAN и GAN Super-Resolution

Super-resolution is the task of producing a larger, sharper image from a smaller one. When an image is shrunk, fine detail such as hair, fabric weave or text strokes is thrown away, and many different high-resolution images shrink to the same small one. A super-resolution model therefore cannot recover the original detail with certainty; it has to choose a plausible version. GAN super-resolution, short for generative adversarial network super-resolution, makes that choice by training the model to produce output that looks like a real photograph. ESRGAN, the Enhanced Super-Resolution Generative Adversarial Network, is the best-known model of this kind, and it typically enlarges images by a factor of four in each direction.

The method works in steps. First, training pairs are made by taking high-resolution images and shrinking them, so the model learns to map small inputs to their known large versions. Second, a generator network, a deep stack of convolutional layers, takes the small image and outputs a large one. Third, its output is scored by several losses: a pixel loss that keeps colors and layout close to the target, a perceptual loss that compares features extracted by a pretrained image-recognition network called VGG, and an adversarial loss. Fourth, the adversarial loss comes from a second network, the discriminator, which is trained at the same time to tell generated images from real ones. As the discriminator improves at spotting smooth, fake-looking output, the generator is pushed to produce crisp textures that pass as real.

The idea came from SRGAN, posted by Christian Ledig and colleagues in September 2016 and presented at CVPR 2017. Before it, most super-resolution models minimized mean squared error against the target, which scores well on PSNR, a pixel-accuracy metric, but produces blurry results because averaging many plausible textures yields a smear. SRGAN replaced that with a perceptual loss plus adversarial loss and reported that human raters in a mean-opinion-score study judged its four-times upscales closer to the originals than any other method tested. ESRGAN, posted in September 2018 by Xintao Wang and colleagues from the Chinese University of Hong Kong, SenseTime and partner institutes, and presented at an ECCV 2018 workshop, improved SRGAN in three ways. It replaced the basic block with a Residual-in-Residual Dense Block and removed batch normalization, it used a relativistic discriminator that judges whether a real image looks more realistic than generated ones on average, and it computed perceptual loss on VGG features before their activation function, which the authors say gives sharper edges and more accurate brightness.

Real-ESRGAN, posted in July 2021 by Wang, Liangbin Xie, Chao Dong and Ying Shan, tackled ESRGAN's biggest practical weakness. ESRGAN was trained on clean images shrunk with a simple bicubic filter, so it handled real photos poorly, because real low-quality images also carry blur, sensor noise, JPEG compression and resizing artifacts. Real-ESRGAN trains only on synthetic data but generates it with a high-order degradation model that applies blur, resizing, noise and compression in randomized sequences, sometimes twice, and it adds a U-Net discriminator with spectral normalization to keep training stable. Its open-source release added specialized anime models, a lightweight general model with adjustable denoising, and portable executables.

The obvious alternatives each make a different trade. Classic interpolation such as bicubic resizing is fast and invents nothing, but looks soft. Models trained only on pixel loss give higher PSNR scores but blurrier results. Blau and Michaeli proved in a CVPR 2018 paper, The Perception-Distortion Tradeoff, that the two goals conflict: below a certain level of average distortion, outputs necessarily become easier to tell apart from real images, and this holds for any distortion measure. GAN models deliberately sit at the realistic end of that trade-off. ESRGAN's authors offer a dial between the two, network interpolation, which blends the weights of a PSNR-trained model and a GAN-trained model so a user can choose how much texture to hallucinate.

The central limitation is that the added detail is invented. A GAN upscaler fills in texture that fits patterns in its training data, so it can draw convincing but wrong eyelashes, letters, bricks or license-plate digits. The risk becomes concrete with faces. In June 2020 a demo of PULSE, a CVPR 2020 face upsampling method that searches a generative model for realistic faces that shrink down to the input, turned a pixelated photo of Barack Obama into a white man's face, as recounted by The Gradient. The PULSE paper itself describes producing images that downscale correctly rather than recovering the original. For that reason, GAN-upscaled images should not be treated as evidence in forensic, medical or legal settings. The Real-ESRGAN paper lists its own failure modes: twisted lines caused by aliasing in building and indoor scenes, GAN artifacts on some images, and real-world degradations outside its training distribution that are not removed and may even be amplified. Tiling adds another, visible seams when large images are processed in tiles, a problem the Real-ESRGAN repository explicitly warns about.

In practice, using an ESRGAN-family model is inexpensive. Pretrained weights are freely available, Real-ESRGAN is released under the BSD-3-Clause license, and its portable executables run on consumer GPUs from all three major vendors. Large images are split into tiles to fit GPU memory. Training a new model is more demanding: ESRGAN trained on the DIV2K, Flickr2K and OutdoorSceneTraining datasets using 128 by 128 pixel high-resolution crops, and GAN training needs careful loss weighting to avoid unstable results. The right choice depends on purpose. For viewing pleasure, such as enlarging artwork or old photos, realistic invented texture is usually the goal; for anything where the exact pixels matter, a fidelity-oriented model or simple interpolation is the honest option.