視覺人工智慧指南

LaMa 解析度-穩健修復

LaMa(大掩模修復)是三星 AI 研究人員在 WACV 2022 上提出的圖像修復模型,它透過使用傅立葉卷積一次查看整個圖像來填充照片中的大孔。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 使用 LaMa 從照片中刪除對象
  5. 現實世界的實施
  6. LaMa 分辨率的未来-稳健修复
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

它僅在 256 x 256 裁剪上進行訓練,但適用於更大的圖像,其小尺寸和開放代碼使其成為許多免費工具中的預設物件刪除引擎。

重點摘要

  1. LaMa 是三星 AI 研究人員的大掩模影像修復模型,於 2021 年 9 月發布,並在 WACV 2022 上發布。
  2. 快速傅立葉卷積為 LaMa 從第一層提供了影像範圍的感受野,這有助於填滿大孔和重複圖案。
  3. 基礎 LaMa-Fourier 有 2700 萬個參數,大約是 CoModGAN 大小的四分之一,但在大多數論文測試中獲得了更好的 FID。
  4. LaMa 在 256 x 256 裁切上進行訓練,但工作解析度要高得多,官方 README 中的解析度最高約為 2K。
  5. LaMa 的填充是看似合理的發明,而不是恢復的像素,並且該模型在強視角下與週期性圖案作鬥爭。

深入探討

Image inpainting is the task of filling in a missing or masked region of a photo so that the result looks natural. The user supplies an image and a mask, a black-and-white image marking which pixels to replace, and the model paints new content into the hole. Uses range from removing a tourist from a landmark shot to repairing scratches. The fill is a plausible guess, not a recovery of what was really there: the original pixels are gone, and the model invents content consistent with its surroundings. LaMa, short for Large Mask Inpainting, is a 2021 model designed for the hardest version of this task, where the hole is large and the image is high resolution.

The paper, by Roman Suvorov, Elizaveta Logacheva, Victor Lempitsky and colleagues at Samsung AI Center Moscow, Samsung Research and the Skolkovo Institute of Science and Technology, was posted to arXiv on September 15, 2021 and published at WACV 2022. Its diagnosis was that earlier inpainting networks lacked an effective receptive field, the area of the input that can influence each output pixel. An ordinary convolution looks only at a small neighborhood, such as 3 by 3 pixels, so a network needs many layers before a pixel in the middle of a large hole can see anything outside it. When the hole is bigger than the receptive field, the network has no context and produces blur or repeated smudges.

LaMa's answer has three parts. The first is the fast Fourier convolution (FFC), introduced by Lu Chi, Borui Jiang and Yadong Mu at NeurIPS 2020. An FFC splits a layer's channels into two branches. The local branch uses ordinary convolutions. The global branch applies a fast Fourier transform, which re-expresses the whole feature map as a sum of waves at different frequencies, processes those frequencies with a small learned layer, and transforms back. Because every frequency component depends on every pixel, the global branch gives each layer an image-wide receptive field from the very first block. This also suits repeating patterns such as brick walls, fences and windows, which show up as strong peaks at particular frequencies.

The second part is the loss, the measure of error that training minimizes. LaMa combines an adversarial loss, in which a separate discriminator network tries to tell filled patches from real ones, with a high receptive field perceptual loss. A perceptual loss compares images through the features of another pretrained network rather than pixel by pixel, so it rewards plausible structure instead of an exact copy of the original. LaMa computes it with a ResNet-50 segmentation network that uses dilated convolutions, which spread out their kernel taps to cover large areas, so the loss itself judges global layout. The third part is training data: masks drawn as wide polygonal strokes or large boxes of random shape, chosen with equal probability, so the model learns to fill big holes rather than thin scratches.

The results justified the design. The base LaMa-Fourier model has 27 million parameters, against 109 million for CoModGAN and 85 million for MADF, the two closest competitors. On the Places scene dataset with wide masks it scored a Fréchet Inception Distance (FID, a measure of how realistic a set of images looks, lower is better) of 2.21 against 5.94 for AOT GAN and 3.76 for MADF, though CoModGAN's 1.82 was better on that one test. On segmentation-shaped masks LaMa scored 5.35 against 6.40 for CoModGAN, and on CelebA-HQ faces with wide masks 6.96 against 24.4. The authors trained a larger Big LaMa with 18 residual blocks and 51 million parameters on 4.5 million Places-Challenge images, using eight NVIDIA V100 GPUs for about 240 hours.

The resolution-robust part of the name refers to generalization. LaMa was trained on 256 by 256 crops but evaluated at up to 1536 by 1536, and the paper found that FFC models degrade much less as resolution grows than models with regular or dilated convolutions. The official README says LaMa generalizes to resolutions up to about 2K. A 2022 CVPR workshop paper by Prakhar Kulshreshtha, Brian Pugh and Salma Jiddi of Geomagical Labs pushed further: it optimizes the network's intermediate features at inference time with a multiscale consistency loss, so coarse, coherent structure from a downscaled pass guides fine detail at full resolution. That refinement was merged into the official code.

Compared with diffusion-based inpainting, such as Stable Diffusion inpainting checkpoints, LaMa is a single forward pass with no text prompt. That makes it fast and predictable for removal, where the goal is to make something disappear into the background. Diffusion models can add new content described in a text prompt, like replacing a bench with a fountain, at the cost of many denoising steps per image. IOPaint reflects this split, using LaMa for erasing and diffusion models such as PowerPaint and SDXL inpainting for replacement and outpainting.

LaMa has clear limits. Its authors note that it usually struggles when strong perspective distorts periodic structures, such as a tiled floor receding into the distance, which is common in internet photos unlike its training data. The Fourier version is about 20% slower than a regular-convolution version of the same model. And because it hallucinates plausible content, it should never be used to claim what a hidden area actually contained, for instance in evidence or documentation.

Running LaMa is inexpensive. The checkpoint is small, inference is one network pass, and IOPaint runs it on a CPU, though a GPU is faster. Training a model from scratch is another matter, as Big LaMa's roughly 1,900 GPU-hours show, and the Places-Challenge training images alone take about 476 GB. The official repository provides pretrained models, Docker scripts for CPU and CUDA, and a mask generator for evaluation.

04工作範例

使用 LaMa 從照片中刪除對象

  1. 選擇一張在相當均勻的背景下包含不需要的物體的照片,例如草地上的標誌。

  2. 建立一個與照片大小相同的遮罩,將物件及其周圍的小邊距繪製為白色,將其他所有內容繪製為黑色。

  3. 使用 pip 安裝 IOPaint 並以 CPU 上的 LaMa 型號啟動它,然後打開它所服務的本地網頁。

  4. 載入照片,刷過對象,然後運行填充;或將 image.png 和 image_mask001.png 放在一個資料夾中,並使用 big-lama 運行官方預測腳本。

  5. 以全解析度放大填滿區域,尋找模糊、重複的斑塊或斷裂的直線。

  6. 重複拍攝對象位於磁磚地板前的透視照片,並比較兩種填充物的品質。

它顯示了什麼

一張乾淨的照片以及簡單和硬盒的直接比較。純色背景應該令人信服地填充,而透視圖塊可能會顯示彎曲或未對齊的線條,這是論文作者所描述的故障模式。

現實世界的實施

在 IOPaint 中刪除不需要的人物、物件、缺陷或浮水印,IOPaint 是 Lama-cleaner 開發商 Sanster 的 Apache-2.0 開源工具,它使用 LaMa 進行刪除,並在 CPU、NVIDIA GPU 或 Apple 晶片上運行。

透過將每個蒙版儲存為 image1.png 旁的 image1_mask001.png 並使用 big-lama 檢查點執行 bin/predict.py,使用官方 advimman/lama 儲存庫批次修復照片資料夾。

使用Geomagical Labs的特徵細化方法來提高高解析度結果,該方法在推理時優化LaMa的內部特徵,並在官方程式碼中透過refine=True啟用。

透過 CoreMLaMa 在 Apple 裝置上執行 LaMa,CoreMLaMa 是官方儲存庫自述文件中列出的 Apple Core ML 格式的社群轉換。

使用 FID 和 LPIPS、原始論文中使用的資料集和指標,對 Places 和 CelebA-HQ 上的 LaMa 的新修復模型進行基準測試。

LaMa 分辨率的未来-稳健修复

The LaMa authors left one question explicitly open in their 2022 paper: whether Fourier convolutions can handle deformed periodic signals, such as patterns seen in strong perspective, where the model usually struggles. High-resolution coherence was a second thread, addressed in 2022 by feature refinement at inference time, which trades extra computation per image for better structure without retraining. The broader direction of inpainting has since moved toward diffusion models that can add content guided by text, and tools such as IOPaint now pair LaMa for removal with diffusion models such as PowerPaint, BrushNet and SDXL inpainting for replacement and outpainting. LaMa's lasting role is as a light, fast removal model and a standard baseline on Places and CelebA-HQ. Open problems for the field include filling holes that hide complex objects or text, measuring realism in ways that match human judgment better than FID, and labeling edited images so that plausible fills are not mistaken for real content.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXiv (WACV 2022)使用傅立葉卷積進行分辨率穩健的大型遮罩修復
  2. GitHubadvimman/喇嘛:LaMa 影像修復
  3. NeurIPS 2020快速傅立葉卷積
  4. arXiv (CVPR 2022 Workshop)特徵細化以改進高解析度影像修復
  5. GitHub桑斯特/IOPaint

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加 LaMa 解析度-穩健修復測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是 LaMa 解析度-穩健修復?

LaMa(大掩模修復)是三星 AI 研究人員在 WACV 2022 上提出的圖像修復模型,它透過使用傅立葉卷積一次查看整個圖像來填充照片中的大孔。它僅在 256 x 256 裁剪上進行訓練,但適用於更大的圖像,其小尺寸和開放代碼使其成為許多免費工具中的預設物件刪除引擎。

賦予 LaMa 圖像全球視野的標誌性建築組件是什麼?

LaMa 使用快速傅立葉卷積 (FFC),其部分在頻域中運行,以在單層中提供全局感受野。

為什麼 LaMa 比許多早期的修復模型更能處理大型面具?

由於 FFC 可以立即看到整個影像,因此即使在非常大的缺失區域中,LaMa 也可以連貫地延續紋理和幾何形狀。

LaMa 論文報告了哪些解析行為?

作者報告說,LaMa 在他們的實驗中泛化到高於訓練的分辨率;性能仍然取決於圖像和掩模。

由於其頻域處理,LaMa 能夠特別出色地處理哪種紋理?

頻域卷積非常適合週期性圖案,因此 LaMa 乾淨利落地繼續重複磚牆和瓷磚地板等紋理。

與大型擴散修復系統相比,LaMa通常如何定位?

LaMa 是一種圍繞 FFC 構建的捲積修復方法,通常在需要高效填充的情況下使用;速度和記憶體取決於實現和硬體。