LaMa: inpainting robusto a la resolución

Image inpainting is the task of filling in a missing or masked region of a photo so that the result looks natural. The user supplies an image and a mask, a black-and-white image marking which pixels to replace, and the model paints new content into the hole. Uses range from removing a tourist from a landmark shot to repairing scratches. The fill is a plausible guess, not a recovery of what was really there: the original pixels are gone, and the model invents content consistent with its surroundings. LaMa, short for Large Mask Inpainting, is a 2021 model designed for the hardest version of this task, where the hole is large and the image is high resolution.

The paper, by Roman Suvorov, Elizaveta Logacheva, Victor Lempitsky and colleagues at Samsung AI Center Moscow, Samsung Research and the Skolkovo Institute of Science and Technology, was posted to arXiv on September 15, 2021 and published at WACV 2022. Its diagnosis was that earlier inpainting networks lacked an effective receptive field, the area of the input that can influence each output pixel. An ordinary convolution looks only at a small neighborhood, such as 3 by 3 pixels, so a network needs many layers before a pixel in the middle of a large hole can see anything outside it. When the hole is bigger than the receptive field, the network has no context and produces blur or repeated smudges.

LaMa's answer has three parts. The first is the fast Fourier convolution (FFC), introduced by Lu Chi, Borui Jiang and Yadong Mu at NeurIPS 2020. An FFC splits a layer's channels into two branches. The local branch uses ordinary convolutions. The global branch applies a fast Fourier transform, which re-expresses the whole feature map as a sum of waves at different frequencies, processes those frequencies with a small learned layer, and transforms back. Because every frequency component depends on every pixel, the global branch gives each layer an image-wide receptive field from the very first block. This also suits repeating patterns such as brick walls, fences and windows, which show up as strong peaks at particular frequencies.

The second part is the loss, the measure of error that training minimizes. LaMa combines an adversarial loss, in which a separate discriminator network tries to tell filled patches from real ones, with a high receptive field perceptual loss. A perceptual loss compares images through the features of another pretrained network rather than pixel by pixel, so it rewards plausible structure instead of an exact copy of the original. LaMa computes it with a ResNet-50 segmentation network that uses dilated convolutions, which spread out their kernel taps to cover large areas, so the loss itself judges global layout. The third part is training data: masks drawn as wide polygonal strokes or large boxes of random shape, chosen with equal probability, so the model learns to fill big holes rather than thin scratches.

The results justified the design. The base LaMa-Fourier model has 27 million parameters, against 109 million for CoModGAN and 85 million for MADF, the two closest competitors. On the Places scene dataset with wide masks it scored a Fréchet Inception Distance (FID, a measure of how realistic a set of images looks, lower is better) of 2.21 against 5.94 for AOT GAN and 3.76 for MADF, though CoModGAN's 1.82 was better on that one test. On segmentation-shaped masks LaMa scored 5.35 against 6.40 for CoModGAN, and on CelebA-HQ faces with wide masks 6.96 against 24.4. The authors trained a larger Big LaMa with 18 residual blocks and 51 million parameters on 4.5 million Places-Challenge images, using eight NVIDIA V100 GPUs for about 240 hours.

The resolution-robust part of the name refers to generalization. LaMa was trained on 256 by 256 crops but evaluated at up to 1536 by 1536, and the paper found that FFC models degrade much less as resolution grows than models with regular or dilated convolutions. The official README says LaMa generalizes to resolutions up to about 2K. A 2022 CVPR workshop paper by Prakhar Kulshreshtha, Brian Pugh and Salma Jiddi of Geomagical Labs pushed further: it optimizes the network's intermediate features at inference time with a multiscale consistency loss, so coarse, coherent structure from a downscaled pass guides fine detail at full resolution. That refinement was merged into the official code.

Compared with diffusion-based inpainting, such as Stable Diffusion inpainting checkpoints, LaMa is a single forward pass with no text prompt. That makes it fast and predictable for removal, where the goal is to make something disappear into the background. Diffusion models can add new content described in a text prompt, like replacing a bench with a fountain, at the cost of many denoising steps per image. IOPaint reflects this split, using LaMa for erasing and diffusion models such as PowerPaint and SDXL inpainting for replacement and outpainting.

LaMa has clear limits. Its authors note that it usually struggles when strong perspective distorts periodic structures, such as a tiled floor receding into the distance, which is common in internet photos unlike its training data. The Fourier version is about 20% slower than a regular-convolution version of the same model. And because it hallucinates plausible content, it should never be used to claim what a hidden area actually contained, for instance in evidence or documentation.

Running LaMa is inexpensive. The checkpoint is small, inference is one network pass, and IOPaint runs it on a CPU, though a GPU is faster. Training a model from scratch is another matter, as Big LaMa's roughly 1,900 GPU-hours show, and the Places-Challenge training images alone take about 476 GB. The official repository provides pretrained models, Docker scripts for CPU and CUDA, and a mask generator for evaluation.