視覺人工智慧指南

ESRGAN 和 GAN 超分辨率

ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) is a 2018 model that enlarges images four times while inventing sharp, realistic-looking texture, using a generator trained against a discriminator.

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. Checking how much detail an upscaler invents
  5. 現實世界的實施
  6. ESRGAN 和 GAN 超解析度的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

It and its successor Real-ESRGAN became standard open-source upscalers, but the detail they add is a plausible guess, not recovered information.

重點摘要

  1. ESRGAN, posted in September 2018, upscales images four times using a generator trained against a relativistic discriminator to produce realistic texture.
  2. ESRGAN improved on 2016's SRGAN with Residual-in-Residual Dense Blocks, no batch normalization, and a perceptual loss on VGG features before activation.
  3. Real-ESRGAN, from 2021, handles real-world photos by training on synthetic images degraded with randomized blur, noise, resizing and JPEG compression.
  4. Blau and Michaeli proved in 2018 that perceptual realism and pixel accuracy conflict, so sharper-looking upscales are not more accurate ones.
  5. Detail added by GAN upscalers is invented from training data, so upscaled images should not be used as forensic or legal evidence.

深入探討

Super-resolution is the task of producing a larger, sharper image from a smaller one. When an image is shrunk, fine detail such as hair, fabric weave or text strokes is thrown away, and many different high-resolution images shrink to the same small one. A super-resolution model therefore cannot recover the original detail with certainty; it has to choose a plausible version. GAN super-resolution, short for generative adversarial network super-resolution, makes that choice by training the model to produce output that looks like a real photograph. ESRGAN, the Enhanced Super-Resolution Generative Adversarial Network, is the best-known model of this kind, and it typically enlarges images by a factor of four in each direction.

The method works in steps. First, training pairs are made by taking high-resolution images and shrinking them, so the model learns to map small inputs to their known large versions. Second, a generator network, a deep stack of convolutional layers, takes the small image and outputs a large one. Third, its output is scored by several losses: a pixel loss that keeps colors and layout close to the target, a perceptual loss that compares features extracted by a pretrained image-recognition network called VGG, and an adversarial loss. Fourth, the adversarial loss comes from a second network, the discriminator, which is trained at the same time to tell generated images from real ones. As the discriminator improves at spotting smooth, fake-looking output, the generator is pushed to produce crisp textures that pass as real.

The idea came from SRGAN, posted by Christian Ledig and colleagues in September 2016 and presented at CVPR 2017. Before it, most super-resolution models minimized mean squared error against the target, which scores well on PSNR, a pixel-accuracy metric, but produces blurry results because averaging many plausible textures yields a smear. SRGAN replaced that with a perceptual loss plus adversarial loss and reported that human raters in a mean-opinion-score study judged its four-times upscales closer to the originals than any other method tested. ESRGAN, posted in September 2018 by Xintao Wang and colleagues from the Chinese University of Hong Kong, SenseTime and partner institutes, and presented at an ECCV 2018 workshop, improved SRGAN in three ways. It replaced the basic block with a Residual-in-Residual Dense Block and removed batch normalization, it used a relativistic discriminator that judges whether a real image looks more realistic than generated ones on average, and it computed perceptual loss on VGG features before their activation function, which the authors say gives sharper edges and more accurate brightness.

Real-ESRGAN, posted in July 2021 by Wang, Liangbin Xie, Chao Dong and Ying Shan, tackled ESRGAN's biggest practical weakness. ESRGAN was trained on clean images shrunk with a simple bicubic filter, so it handled real photos poorly, because real low-quality images also carry blur, sensor noise, JPEG compression and resizing artifacts. Real-ESRGAN trains only on synthetic data but generates it with a high-order degradation model that applies blur, resizing, noise and compression in randomized sequences, sometimes twice, and it adds a U-Net discriminator with spectral normalization to keep training stable. Its open-source release added specialized anime models, a lightweight general model with adjustable denoising, and portable executables.

The obvious alternatives each make a different trade. Classic interpolation such as bicubic resizing is fast and invents nothing, but looks soft. Models trained only on pixel loss give higher PSNR scores but blurrier results. Blau and Michaeli proved in a CVPR 2018 paper, The Perception-Distortion Tradeoff, that the two goals conflict: below a certain level of average distortion, outputs necessarily become easier to tell apart from real images, and this holds for any distortion measure. GAN models deliberately sit at the realistic end of that trade-off. ESRGAN's authors offer a dial between the two, network interpolation, which blends the weights of a PSNR-trained model and a GAN-trained model so a user can choose how much texture to hallucinate.

The central limitation is that the added detail is invented. A GAN upscaler fills in texture that fits patterns in its training data, so it can draw convincing but wrong eyelashes, letters, bricks or license-plate digits. The risk becomes concrete with faces. In June 2020 a demo of PULSE, a CVPR 2020 face upsampling method that searches a generative model for realistic faces that shrink down to the input, turned a pixelated photo of Barack Obama into a white man's face, as recounted by The Gradient. The PULSE paper itself describes producing images that downscale correctly rather than recovering the original. For that reason, GAN-upscaled images should not be treated as evidence in forensic, medical or legal settings. The Real-ESRGAN paper lists its own failure modes: twisted lines caused by aliasing in building and indoor scenes, GAN artifacts on some images, and real-world degradations outside its training distribution that are not removed and may even be amplified. Tiling adds another, visible seams when large images are processed in tiles, a problem the Real-ESRGAN repository explicitly warns about.

In practice, using an ESRGAN-family model is inexpensive. Pretrained weights are freely available, Real-ESRGAN is released under the BSD-3-Clause license, and its portable executables run on consumer GPUs from all three major vendors. Large images are split into tiles to fit GPU memory. Training a new model is more demanding: ESRGAN trained on the DIV2K, Flickr2K and OutdoorSceneTraining datasets using 128 by 128 pixel high-resolution crops, and GAN training needs careful loss weighting to avoid unstable results. The right choice depends on purpose. For viewing pleasure, such as enlarging artwork or old photos, realistic invented texture is usually the goal; for anything where the exact pixels matter, a fidelity-oriented model or simple interpolation is the honest option.

04工作範例

Checking how much detail an upscaler invents

  1. Pick a sharp photo with readable small text, for example a 1,024 by 1,024 pixel image of a street sign.

  2. Shrink it four times to 256 by 256 pixels with bicubic resizing, so the text becomes unreadable.

  3. Upscale the small image four times with plain bicubic resizing and save the result.

  4. Upscale the same small image with Real-ESRGAN's RealESRGAN_x4plus model and save that result.

  5. Compare both outputs side by side with the original, zooming in on the text and fine textures.

  6. Compute PSNR for each output against the original, using any image library's PSNR function.

它顯示了什麼

The Real-ESRGAN output usually looks much sharper while the bicubic output looks soft, yet letters in the GAN output may be shaped wrongly. The PSNR comparison shows whether the sharper image is actually closer to the original, illustrating the perception-distortion trade-off.

現實世界的實施

Upscaling photos and artwork with the open-source Real-ESRGAN project, whose default RealESRGAN_x4plus model enlarges images four times and whose GitHub repository has over 37,000 stars.

Enlarging anime illustrations and anime video with Real-ESRGAN's dedicated RealESRGAN_x4plus_anime_6B and realesr-animevideov3 models, tuned for flat colors and line art.

Running upscaling on ordinary Windows, Linux or macOS machines with Real-ESRGAN's portable ncnn-vulkan executables, which work on Intel, AMD and Nvidia GPUs without installing CUDA or PyTorch.

Restoring faces in old or low-quality photos by combining Real-ESRGAN for the background with its --face_enhance option, which calls the separate GFPGAN face restoration model.

Benchmarking perceptual quality: ESRGAN won region 3 of the PIRM2018-SR Challenge, a competition that ranked super-resolution methods by a no-reference perceptual index rather than pixel accuracy.

ESRGAN 和 GAN 超解析度的未來

The core tension in this field, formalized in Blau and Michaeli's 2018 perception-distortion result, has not gone away: any upscaler that looks more realistic must, past a point, drift further from the true image on average. Research since ESRGAN has largely focused on the input side, as Real-ESRGAN did in 2021 by simulating realistic degradations, and on the generative engine, and on the generative engine itself. Each new method faces the same questions about how much invented detail is acceptable and how to measure quality when no single metric captures both fidelity and realism. The social questions raised by the PULSE episode in June 2020 also remain open. Face-focused upscalers inherit biases from their training data, and outputs can be mistaken for recovered evidence. Directions discussed in that debate, such as model cards documenting training data and intended use, help users judge a released model, but there is no technical fix that lets a model know detail that was never captured. Users and institutions will continue to need policies on when enhanced images may be used and how they must be labeled.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXiv / ECCV 2018 Workshops (Wang et al.)ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
  2. arXiv / CVPR 2017 (Ledig et al.)Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
  3. arXiv (Wang, Xie, Dong, Shan)Real-ESRGAN:使用純合成資料訓練真實世界盲超分辨率
  4. GitHubxinntao/Real-ESRGAN
  5. arXiv / CVPR 2018 (Blau, Michaeli)The Perception-Distortion Tradeoff
  6. arXiv / CVPR 2020 (Menon et al.)PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models
  7. The GradientLessons from the PULSE Model and Discussion

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the ESRGAN and GAN Super-Resolution quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是 ESRGAN 和 GAN 超解析度?

ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) is a 2018 model that enlarges images four times while inventing sharp, realistic-looking texture, using a generator trained against a discriminator. It and its successor Real-ESRGAN became standard open-source upscalers, but the detail they add is a plausible guess, not recovered information.

ESRGAN 中的「GAN」指的是什麼?

GAN 代表生成對抗網絡,是生成器和判別器在訓練期間競爭的設置。

ESRGAN 生成器主要使用什麼建構塊?

ESRGAN 的生成器堆疊 Residual-in-Residual Dense Blocks (RRDB),即密集連接的殘差單元,作為其核心結構。

為什麼 ESRGAN 的作者從生成器中刪除了批量歸一化?

作者發現批量歸一化在超解析度輸出中產生了令人不快的偽影,因此他們將其從 RRDB 區塊中刪除。

與僅像素損失方法相比,ESRGAN 的主要權衡是什麼?

即使單一像素值與地面真實值不同,對抗性損失也會將輸出推向逼真的紋理。

ESRGAN 的感知 (VGG) 損失在哪裡測量?

ESRGAN 在啟動函數之前計算 VGG 特徵圖上的感知損失,作者發現這給出了更清晰的結果。