概述
它發佈於 CVPR 2021,成為最廣泛使用的工具之一,用於修復舊照片和修復人工智慧生成的圖像中的臉部,同時也說明了捏造細節的風險。
重點摘要
- GFPGAN 使用預先訓練的 StyleGAN2 臉部產生器的特徵作為先驗,在一次前向傳遞中恢復退化的臉部。
- 通道分割空間特徵變換層使用輸入資訊調製一半的 GAN 特徵,以平衡真實性和保真度。
- CVPR 2021 論文在 70,000 張影像 FFHQ 資料集上進行訓練,具有合成模糊、下取樣、雜訊和 JPEG 損壞。
- 在 CelebA-Test 上,GFPGAN 報告了比較方法中最佳的 LPIPS、FID、NIQE 和身份距離。
- 修復後的臉孔包含看似合理的虛構細節,因此不應將其視為真人的準確記錄。
深入探討
Blind face restoration means recovering a clean, detailed face from a damaged photo when the exact damage is unknown. A real old snapshot might be blurred, shrunk, noisy and saved as a heavily compressed JPEG, all at once. GFPGAN, short for Generative Facial Prior GAN, attacks this by combining two sources of information: what can still be seen in the input, and what a separate network already knows about how faces look. That second source is the generative facial prior, taken from StyleGAN2, a generative adversarial network (GAN) trained to produce photorealistic faces.
The method was introduced in 'Towards Real-World Blind Face Restoration with Generative Facial Prior' by Xintao Wang, Yu Li, Honglun Zhang and Ying Shan of Tencent's Applied Research Center, posted to arXiv on 11 January 2021 and accepted at CVPR 2021. Earlier approaches leaned on geometric priors such as facial landmarks or parsing maps, which are hard to estimate from a badly degraded input, or on reference photos of the same person, which are rarely available. GAN inversion methods such as PULSE searched for a StyleGAN latent code that matched the input, which was slow and often produced a convincing face that was not the same person. GFPGAN set out to keep the realism of a GAN prior while staying faithful to the input, in one forward pass.
The pipeline has three stages. First, a degradation removal module, a U-Net with seven downsampling and seven upsampling steps, cleans the input and produces two outputs: multi-resolution spatial features and a compact latent feature. Second, the latent feature is mapped through small fully connected layers to intermediate latent codes for a StyleGAN2 generator pretrained at 512 by 512 pixels, which generates rich facial features. Third, Channel-Split Spatial Feature Transform layers, called CS-SFT, blend the two. At each resolution, the input's spatial features predict a scale and shift that modulate half of the StyleGAN2 feature channels, while the other half pass through untouched. The modulated half keeps the result tied to the real face; the untouched half preserves the GAN's texture and realism.
Training combines several losses. A reconstruction loss pairs pixel L1 distance with a perceptual loss from a VGG-19 network, and is applied at several pyramid levels inside the degradation removal module. An adversarial loss pushes the whole face to look real. Small local discriminators judge crops of the left eye, right eye and mouth, with a Gram-matrix style loss to sharpen fine texture. An identity preserving loss compares ArcFace face-recognition embeddings of the output and the ground truth, so the restored face still reads as the same person. The paper trained on the 70,000-image FFHQ dataset resized to 512 pixels, synthesizing damaged inputs by blurring, downsampling by a factor of 1 to 8, adding Gaussian noise and JPEG compressing at quality 60 to 100. Training ran for 800,000 iterations on four Nvidia Tesla P40 GPUs.
On the synthetic CelebA-Test benchmark of 3,000 images, the paper reports GFPGAN at an LPIPS of 0.3646, FID of 42.62 and NIQE of 4.077, the best perceptual scores among the methods compared, including PSFRGAN, DFDNet, HiFaceGAN and PULSE. LPIPS measures perceptual distance to the ground truth, FID compares image statistics to real faces, and NIQE scores naturalness without a reference; lower is better for all three. Its identity distance of 34.60 was also the lowest, while its PSNR of 25.08 dB was slightly below a fine-tuned DeblurGANv2 at 25.91, a reminder that pixel-accurate and realistic-looking are different goals. The authors also tested on real low-quality sets: LFW-Test, CelebChild-Test and 407 faces from 188 web photos.
The open-source release grew after the paper. The GitHub repository, under the Apache 2.0 license, lists V1 as the paper model with colorization, V1.2 as sharper but sometimes unnatural with a made-up look, V1.3 as more natural with slight identity change, and V1.4 as adding slightly more detail and better identity. A 'clean' architecture removed the need for custom CUDA extensions, which made installation much easier. Non-face regions are enlarged by Real-ESRGAN, a sibling project from the same team. The repository showed about 37,700 GitHub stars when checked for this article.
Compared with alternatives, GFPGAN sits between older prior-based restorers and newer approaches. CodeFormer, published at NeurIPS 2022 by Shangchen Zhou and colleagues, replaces the continuous GAN prior with a learned discrete codebook and a transformer that predicts codes, and offers a fidelity weight to trade realism against faithfulness. Its authors report better robustness under heavy damage. Diffusion-based face restorers are a further line of work. GFPGAN's advantages are speed and simplicity: one forward pass, no per-image optimization, and modest hardware needs.
The limitations matter. The paper notes that under very severe degradation, restored details twist into artifacts, and very large head poses yield unnatural faces, because synthetic training damage differs from real-world damage. Grayscale inputs can produce biased skin color since there is no color information to anchor it. More fundamentally, any generative restorer fills missing detail with plausible guesses. The output can look sharp yet differ from the real person, so GFPGAN results should not be treated as evidence of what someone looked like, for example in forensic or journalistic use. Running it locally needs a PyTorch install and benefits from a GPU; hosted runs on Replicate cost fractions of a cent.
04工作範例
使用 GFPGAN 恢復掃描的家庭照片
複製 TencentARC/GFPGAN 儲存庫,安裝其 Python 需求以及 realesrgan 套件,並下載 README 中連結的 V1.4 模型權重。
以合理的解析度掃描一張舊照片,並將其放入輸入資料夾中。
對該資料夾運行推理腳本,將版本標誌設為 1.4,放大係數設為 2,並保留 Real-ESRGAN 作為背景放大器。
打開結果資料夾,其中包含裁剪後的輸入面孔、恢復的面孔、並排比較和完整恢復的圖像。
將每張修復後的臉部與原始臉部進行比較,尋找眼睛形狀、牙齒、皮膚紋理或表情的變化。
重複版本 1.3,並注意哪個輸出可以更好地保持人物的相似度。
它顯示了什麼
學習者將獲得一張放大兩倍的照片,其中包含更清晰的臉部和由 Real-ESRGAN 放大的背景。檢查對比影像可以看出GFPGAN在哪裡發明了細節,這是分享或存檔結果之前的關鍵判斷。
現實世界的實施
AUTOMATIC1111 Stable Diffusion Web UI 提供 GFPGAN 或 CodeFormer 作為內建臉部修復選項,並使用滑桿控制修復在產生的肖像上的可見程度。
將主機 tencentarc/gfpgan 複製為按使用付費 API,在 Nvidia L40S GPU 上列出大約 5 秒內的典型預測,每次運行約 0.0045 美元。
家族歷史學家使用 GFPGAN 的推論腳本搭配 Real-ESRGAN 背景上採樣器,將掃描印刷品進行處理,以在放大整張照片兩倍的同時提升臉部清晰度。
研究人員在發表更新的臉部修復器(如 CodeFormer)時,會以 GFPGAN 的 CelebA-Test 結果作為基準,因為它對嚴重降質的韌性更佳。
新人不需要安裝任何東西,就能透過 Hugging Face 的 Gradio 示範與 Colab 筆記本(從官方倉庫連結)試用 GFPGAN,上傳照片並下載修復結果。
GFPGAN人臉修復的未來
GFPGAN 這篇論文將其下一步命名為:從真實影像學習降質與資料分佈,而非僅僅依賴合成影像,因為不匹配會在嚴重損壞與大角度姿勢下產生瑕疵。此後的工作探索了其他先驗。CodeFormer(NeurIPS 2022)使用離散碼本與 Transformer,並提供明確的保真度控制;擴散模型則越來越多地被用作修復的先驗,因為它們能很好地模擬細緻紋理。未解決的問題包括可靠測量身份保留、避免產生改變人物外貌的幻覺細節、保持影片幀間結果一致,以及降低人口統計偏差,論文本身已指出灰階輸入存在此問題,並呼籲使用更具多樣性與平衡的訓練資料。隨著修復工具進入消費者相片應用,修復影像的來源與標註可能會受到更多關注,儘管平台如何處理此問題尚未決定。
付諸實踐
為什麼這很重要
- 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
- 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
- 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。
留意
- 如果出處不明,肖像權和同意可能會成為法律風險。
- 模型表現可能因光照、人口統計和環境的不同而有所不同。
- 除非監控置信閾值,否則誤報可能會被忽略。
後續步驟
- 定義精確度、召回率和錯誤成本的接受標準。
- 使用符合實際生產條件的數據進行測試。
- 為低置信度或高影響力的預測添加人工審核。
- 追蹤模型漂移並在相機或資料集變更後重新驗證。
資料來源與延伸閱讀
- arXiv (CVPR 2021)利用生成臉部先驗實現現實世界的盲人臉部恢復
- GitHub (Tencent ARC Lab)騰訊ARC/GFPGAN
- arXiv (NeurIPS 2022)使用 Codebook Lookup Transformer 實現穩健盲人臉恢復
- arXivReal-ESRGAN:使用純合成資料訓練真實世界盲超分辨率
- Replicate騰訊/gfpgan
- GitHub (AUTOMATIC1111)功能(stable-diffusion-webui wiki)
不斷探索
常見問題
什麼是GFPGAN人臉修復?
GFPGAN 是一個由 Tencent 的 ARC Lab 開源的模型,能在單次通過時修復模糊、噪點、低解析度或壓縮的臉部,並透過預訓練的 StyleGAN2 臉部生成器借用細節。於 CVPR 2021 發表後,它成為修復舊照片與 AI 生成影像中臉部的最廣泛使用工具之一,同時也說明了創造細節的風險。
什麼預訓練模型提供了 GFPGAN 的「生成臉部先驗」?
GFPGAN 重新使用預先訓練的 StyleGAN2 作為固定解碼器,該解碼器已經對真實面孔進行了編碼。
GFPGAN 的 CS-SFT 層的作用是什麼?
通道分割的空間特徵轉換層結合了預先訓練的臉部先驗,同時使用輸入特徵來指導結果。
GFPGAN 論文報告了哪些關於推理的內容?
與使用昂貴的推理時間優化的圖像特定 GAN 反演不同,所提出的模型在單次前向傳遞中恢復細節。
哪種配對模型通常與 GFPGAN 配對來處理後台?
GFPGAN 可以恢復人臉,而 Real-ESRGAN 在許多捆綁工具中可以放大並清理周圍的背景。
在訓練過程中,GFPGAN 不需要什麼?
GFPGAN 利用通用臉部先驗,因此不需要每個被恢復個體的匹配高品質參考。