視覺人工智慧指南

SwinIR 變壓器恢復

SwinIR 是由蘇黎世聯邦理工學院(ETH Zurich)於 2021 年 8 月發表的影像修復網路,它利用 Swin Transformer 的位移視窗注意力機制來進行影像放大、去噪以及去除 JPEG 壓縮偽影。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 追蹤 SwinIR 的 4 倍放大過程
  5. 現實世界的實施
  6. SwinIR 變壓器恢復的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

它的重要性在於證明了一個約 1,200 萬參數的輕量級 Transformer,在標準修復基準測試中能超越更大的卷積模型與 Transformer 模型,並成為常見的放大器以及後續研究的基準。

重點摘要

  1. SwinIR 由蘇黎世聯邦理工學院的研究人員於 2021 年 8 月發布,將 Swin Transformer 的視窗注意力機制應用於超解析度、去噪和 JPEG 偽影減少。
  2. 其預設模型堆疊了 6 個殘差 Swin Transformer 區塊,每個區塊包含 6 層,並使用 8x8 的注意力視窗與 180 個通道。
  3. SwinIR 在超解析度任務上擊敗了 Transformer 模型 IPT,且參數僅為 1,180 萬,遠低於 IPT 的 1.155 億。
  4. 參數較少並不代表推論速度更快:論文指出,處理一張 1,024 x 1,024 的影像約需 1.1 秒,而 CNN 模型 RCAN 僅需約 0.2 秒。
  5. 包括 Swin2SR 和 HAT 在內的後繼模型均直接基於 SwinIR 構建,並報告了優於 SwinIR 的結果。

深入探討

SwinIR, short for Swin Image Restoration, is a neural network that takes a damaged or low-resolution image and outputs a cleaner or larger version. Image restoration covers several problems: super-resolution, which enlarges an image while inventing plausible fine detail; denoising, which removes random grain; and JPEG compression artifact reduction, which removes the blocky patterns and color smearing that heavy compression leaves behind. SwinIR handles all of them with one architecture trained separately for each task. It was introduced by Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool and Radu Timofte of the Computer Vision Lab at ETH Zurich, with KU Leuven, in a paper posted to arXiv on August 23, 2021.

Before SwinIR, the best restoration models were convolutional neural networks (CNNs), which slide small learned filters across an image. The SwinIR paper points to two weaknesses of convolution for this job: the same filter is applied to every region regardless of content, and each filter sees only a small neighborhood, so long-range patterns are hard to use. Transformers, which use self-attention to let every position weigh information from other positions based on content, address both, but full self-attention over every pixel pair is far too expensive for large images. The earlier transformer model IPT also needed more than 115.5 million parameters and pretraining on ImageNet.

SwinIR's answer is the Swin Transformer, a vision backbone posted to arXiv in March 2021 by Ze Liu and colleagues. Swin computes attention only inside small non-overlapping windows, which keeps cost linear in image size rather than quadratic, and it alternates between a regular window grid and one shifted by half a window so that information flows across window borders from one layer to the next. SwinIR adapts this to restoration in three stages. A single 3 by 3 convolution extracts shallow features from the input. A deep feature extraction module then applies a stack of residual Swin Transformer blocks (RSTBs); each contains several Swin Transformer layers followed by a convolution and a skip connection that adds the block's input back to its output. Finally, a reconstruction module combines shallow and deep features and, for super-resolution, enlarges them with a sub-pixel convolution, a layer that rearranges extra feature channels into extra pixels.

The default configuration in the paper uses 6 RSTBs, 6 Swin layers per block, an 8 by 8 attention window, 180 feature channels and 6 attention heads, about 11.8 million parameters for classical super-resolution. A lightweight version cuts this to 4 blocks and 60 channels, under 900,000 parameters. Super-resolution models are trained with an L1 loss, the average absolute pixel difference from the ground truth; denoising and JPEG models use the closely related Charbonnier loss. For real-world photos, where no perfect ground truth exists, the model is retrained on images degraded with the BSRGAN degradation model and trained with a mix of pixel, adversarial (GAN) and perceptual losses to look sharper.

On benchmarks, the paper reports state-of-the-art results on all three task families. In classical super-resolution trained on the DIV2K dataset, SwinIR's largest gain over prior methods was 0.26 dB of peak signal-to-noise ratio (PSNR, a pixel-level fidelity score in decibels) on Manga109 at 4x, and up to 0.47 dB more when trained on DIV2K plus Flickr2K. It beat IPT while using 11.8 million parameters instead of 115.5 million. In lightweight super-resolution it gained up to 0.53 dB at a similar size to competitors. In JPEG artifact reduction it gained at least 0.11 dB and 0.07 dB on two test sets over DRUNet with 11.5 million parameters against DRUNet's 32.7 million. The abstract summarizes the gains as 0.14 to 0.45 dB with up to 67% fewer parameters.

These gains came with costs. The paper reports that on a 1,024 by 1,024 image, the CNN model RCAN took about 0.2 seconds, SwinIR about 1.1 seconds and IPT about 4.5 seconds. The SwinIR repository gives a closer comparison on Manga109 at 4x: SwinIR scored 31.67 dB against RCAN's 31.22 dB with fewer parameters (11.9 million against 15.6 million) and slightly fewer operations, but ran about three times slower on a 256 by 256 input (0.539 against 0.180 seconds) and used more memory. Attention over windows is less hardware-friendly than plain convolution, so fewer parameters does not mean faster inference.

SwinIR also shares the limits of all restoration models. Each model is trained for one task and one degradation level, such as 4x upscaling or noise level 25, and does poorly on inputs that do not match. PSNR rewards blurry averages, so the highest-scoring models can look soft, while GAN-trained real-world models look sharper but can invent textures that were never in the scene. That makes upscaled images unsuitable as evidence of fine detail, for example in forensic or medical settings. Later analysis in the HAT paper (2022) found that window-based super-resolution transformers use only a limited spatial range of the input, which motivated designs that widen it.

In practice SwinIR is easy to use. The official code and pretrained models are released under the Apache 2.0 license, and the repository reports training a classical model in about 1.8 days on eight Nvidia RTX 2080 Ti GPUs for 500,000 iterations. For most people the simplest route is a tool that bundles it, such as the AUTOMATIC1111 web UI's upscaler menu, which lists SwinIR and Swin2SR alongside Real-ESRGAN and ESRGAN.

04工作範例

追蹤 SwinIR 的 4 倍放大過程

  1. 從一張應成為 256乘256 的 64乘64 像素彩色影像開始。

  2. 一個 3乘3 的摺積將 3 個顏色通道轉換為 180 個特徵通道,尺寸仍維持 64乘64。

  3. 將 64乘64 的特徵圖分割成 8乘8 的視窗:64 個視窗,每個視窗包含 64 個 Token,並在每個視窗內計算注意力。

  4. 在下一層中,在分割前將地圖平移 4 個像素,如此一來,原本位於視窗邊緣的 Token 現在就能與其鄰居共享一個視窗。

  5. 在經過 6 個包含 6 層的區塊後,透過長捷徑連線將淺層特徵加回。

  6. 子像素摺積將額外的通道重新排列成像素,產生一個具有 3 個顏色通道的 256乘256 影像。

  7. 使用 PSNR 將結果與真實的高解析度影像進行比較,並以肉眼檢查邊緣與文字。

它顯示了什麼

學習者可以了解 SwinIR 如何透過在 64 個 token 的視窗中運作來降低注意力機制的成本、位移(shifting)如何連接這些視窗,以及上採樣(upsampling)如何僅在最後階段執行。比雙三次插值(bicubic enlargement)更高的 PSNR 證實了該模型確實能恢復細節,而視覺檢查則能確認這些細節是否合理。

現實世界的實施

創意工具中的照片放大:AUTOMATIC1111 Stable Diffusion 網頁 UI 在其「額外功能」(Extras)標籤頁中,將 SwinIR 及其後繼者 Swin2SR 列為神經網路放大器,用於放大生成的影像。

真實世界的超解析度:SwinIR 儲存庫提供了針對真實照片訓練的 4 倍 GAN 模型,這些模型使用 BSRGAN 退化模型進行訓練,該模型模擬了真實低品質影像中常見的模糊、雜訊和壓縮。

JPEG 清理:SwinIR 的偽影消除模型針對 JPEG 品質等級 10、20、30 和 40 發布,包含灰階與彩色版本,並使用 7x7 的注意力視窗,這是因為 JPEG 採用 8x8 的區塊。

去噪:獨立的 SwinIR 模型可去除等級 15、25 和 50 的高斯雜訊,論文指出在 Urban100 測試集上,其表現比更大的 DRUNet 模型高出 0.3 dB。

研究基準:後續的修復網路如 Swin2SR(ECCV 2022 工作坊)和 HAT(CVPR 2023)皆以 SwinIR 作為對照基準來報告其結果,使其成為一個標準的參考點。

SwinIR 變壓器恢復的未來

SwinIR's design was quickly extended. Swin2SR, posted in September 2022 by Marcos Conde and colleagues including SwinIR coauthor Radu Timofte, swapped in Swin Transformer V2 to address training instability and data hunger, reported faster convergence and better results than SwinIR, and placed among the top five entries in the AIM 2022 challenge on super-resolution of compressed images and video. HAT, posted in May 2022 and accepted to CVPR 2023, combined channel attention with window attention and an overlapping cross-attention module, and reported gains of more than 1 dB over the prior state of the art. Open problems remain. Restoration models trained on synthetic degradations still struggle when real damage differs from the training recipe, and there is no single metric that captures both fidelity and perceived sharpness: PSNR favors smooth results, while GAN-based and diffusion-based restorers favor plausible detail that may not be real. Running window-attention models at high resolution on phones and in real-time video is still harder than running CNNs of similar quality. The balance between faithful reconstruction and realistic invention remains an active research question rather than a settled one.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXivSwinIR:使用 Swin Transformer 進行影像修復
  2. GitHubSwinIR 官方儲存庫
  3. arXivSwin Transformer:使用位移視窗的階層式視覺 Transformer
  4. arXiv (ECCV 2022 Workshops)Swin2SR:用於壓縮影像超解析度與修復的 SwinV2 Transformer
  5. arXiv (CVPR 2023)在影像超解析度 Transformer 中啟用更多像素
  6. GitHubStable Diffusion 網頁 UI

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加 SwinIR Transformer 修復測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是 SwinIR Transformer 修復?

SwinIR 是由蘇黎世聯邦理工學院(ETH Zurich)於 2021 年 8 月發表的影像修復網路,它利用 Swin Transformer 的位移視窗注意力機制來進行影像放大、去噪與去除 JPEG 偽影。它的重要性在於證明了一個約 1200 萬參數的輕量級 Transformer 可以在標準修復基準測試中擊敗更大的卷積模型與 Transformer 模型,並成為常見的放大器與後續研究的基準。

SwinIR 是基於什麼變壓器架構?

SwinIR 採用 Swin Transformer 的分層、基於視窗的設計來進行影像復原。

SwinIR的核心注意力機制是什麼?

SwinIR 在本機視窗內使用自註意力機制,在層之間切換以混合跨視窗的資訊。

SwinIR 的深層功能塊叫什麼?

深度特徵提取階段堆疊了 Residual Swin Transformer 區塊,每個區塊都有 Swin 層、一個卷積和一個殘差連接。

為什麼基於視窗的注意力比全局注意力更有效?

在固定大小的視窗內計算注意力使得成本與影像區域成線性比例,避免了全局注意力的二次爆炸。

SwinIR 修復架構的三個主要階段是什麼?

SwinIR 論文描述了淺層特徵提取、使用 RSTB 的深層特徵提取,以及重建模組。