Amabwiriza ya AI

Icyemezo cya LaMa-Igishushanyo gikomeye

LaMa (Inkingi nini ya Mask) ni ishusho yerekana ishusho yakozwe nabashakashatsi ba Samsung AI, yerekanwe kuri WACV 2022, yuzuza ibyobo binini mumafoto ukoresheje ibyemezo bya Fourier kugirango ubone ishusho yose icyarimwe.

Umva iki gitaboAI-byavuzwe · ~8 min

Kuriyi page9 min soma
  1. Incamake
  2. Ibyingenzi byingenzi
  3. Kwibira cyane
  4. Kuraho ikintu kumafoto hamwe na LaMa
  5. Gushyira mu bikorwa Isi
  6. Ejo hazaza h'icyemezo cya LaMa-Igishushanyo gikomeye
  7. Kubishyira mubikorwa
  8. Inkomoko no gusoma
  9. Komeza Ubushakashatsi
  10. Ibibazo bikunze kubazwa

Incamake

Yahuguwe gusa kuri 256 kubihingwa 256, ikora kumashusho manini cyane, kandi ubunini bwayo buto hamwe na code ifunguye byatumye moteri isanzwe ikuraho ibintu mubikoresho byinshi byubusa.

Ibyingenzi byingenzi

  1. LaMa ni ishusho nini ya mask yerekana ishusho yakozwe n'abashakashatsi ba Samsung AI, yashyizwe ahagaragara muri Nzeri 2021 kandi isohoka kuri WACV 2022.
  2. Imyanzuro yihuse ya Fourier iha LaMa ishusho yagutse yakira kuva murwego rwayo rwa mbere, ifasha kuzuza ibyobo binini no gusubiramo ibishushanyo.
  3. Shingiro LaMa-Fourier ifite ibipimo bya miliyoni 27, hafi kimwe cya kane kingana na CoModGAN, nyamara yatsinze FID nziza mubizamini byinshi byimpapuro.
  4. LaMa itoza ku bihingwa 256 ku bihingwa 256 ariko ikora ku myanzuro ihanitse cyane, ibyo README yemewe ishyira hafi 2K.
  5. LaMa yuzuza ni ibintu byemewe byavumbuwe, ntabwo byagaruwe pigiseli, kandi icyitegererezo kirwana nuburyo bwigihe cyerekezo gikomeye.

Kwibira cyane

Image inpainting is the task of filling in a missing or masked region of a photo so that the result looks natural. The user supplies an image and a mask, a black-and-white image marking which pixels to replace, and the model paints new content into the hole. Uses range from removing a tourist from a landmark shot to repairing scratches. The fill is a plausible guess, not a recovery of what was really there: the original pixels are gone, and the model invents content consistent with its surroundings. LaMa, short for Large Mask Inpainting, is a 2021 model designed for the hardest version of this task, where the hole is large and the image is high resolution.

The paper, by Roman Suvorov, Elizaveta Logacheva, Victor Lempitsky and colleagues at Samsung AI Center Moscow, Samsung Research and the Skolkovo Institute of Science and Technology, was posted to arXiv on September 15, 2021 and published at WACV 2022. Its diagnosis was that earlier inpainting networks lacked an effective receptive field, the area of the input that can influence each output pixel. An ordinary convolution looks only at a small neighborhood, such as 3 by 3 pixels, so a network needs many layers before a pixel in the middle of a large hole can see anything outside it. When the hole is bigger than the receptive field, the network has no context and produces blur or repeated smudges.

LaMa's answer has three parts. The first is the fast Fourier convolution (FFC), introduced by Lu Chi, Borui Jiang and Yadong Mu at NeurIPS 2020. An FFC splits a layer's channels into two branches. The local branch uses ordinary convolutions. The global branch applies a fast Fourier transform, which re-expresses the whole feature map as a sum of waves at different frequencies, processes those frequencies with a small learned layer, and transforms back. Because every frequency component depends on every pixel, the global branch gives each layer an image-wide receptive field from the very first block. This also suits repeating patterns such as brick walls, fences and windows, which show up as strong peaks at particular frequencies.

The second part is the loss, the measure of error that training minimizes. LaMa combines an adversarial loss, in which a separate discriminator network tries to tell filled patches from real ones, with a high receptive field perceptual loss. A perceptual loss compares images through the features of another pretrained network rather than pixel by pixel, so it rewards plausible structure instead of an exact copy of the original. LaMa computes it with a ResNet-50 segmentation network that uses dilated convolutions, which spread out their kernel taps to cover large areas, so the loss itself judges global layout. The third part is training data: masks drawn as wide polygonal strokes or large boxes of random shape, chosen with equal probability, so the model learns to fill big holes rather than thin scratches.

The results justified the design. The base LaMa-Fourier model has 27 million parameters, against 109 million for CoModGAN and 85 million for MADF, the two closest competitors. On the Places scene dataset with wide masks it scored a Fréchet Inception Distance (FID, a measure of how realistic a set of images looks, lower is better) of 2.21 against 5.94 for AOT GAN and 3.76 for MADF, though CoModGAN's 1.82 was better on that one test. On segmentation-shaped masks LaMa scored 5.35 against 6.40 for CoModGAN, and on CelebA-HQ faces with wide masks 6.96 against 24.4. The authors trained a larger Big LaMa with 18 residual blocks and 51 million parameters on 4.5 million Places-Challenge images, using eight NVIDIA V100 GPUs for about 240 hours.

The resolution-robust part of the name refers to generalization. LaMa was trained on 256 by 256 crops but evaluated at up to 1536 by 1536, and the paper found that FFC models degrade much less as resolution grows than models with regular or dilated convolutions. The official README says LaMa generalizes to resolutions up to about 2K. A 2022 CVPR workshop paper by Prakhar Kulshreshtha, Brian Pugh and Salma Jiddi of Geomagical Labs pushed further: it optimizes the network's intermediate features at inference time with a multiscale consistency loss, so coarse, coherent structure from a downscaled pass guides fine detail at full resolution. That refinement was merged into the official code.

Compared with diffusion-based inpainting, such as Stable Diffusion inpainting checkpoints, LaMa is a single forward pass with no text prompt. That makes it fast and predictable for removal, where the goal is to make something disappear into the background. Diffusion models can add new content described in a text prompt, like replacing a bench with a fountain, at the cost of many denoising steps per image. IOPaint reflects this split, using LaMa for erasing and diffusion models such as PowerPaint and SDXL inpainting for replacement and outpainting.

LaMa has clear limits. Its authors note that it usually struggles when strong perspective distorts periodic structures, such as a tiled floor receding into the distance, which is common in internet photos unlike its training data. The Fourier version is about 20% slower than a regular-convolution version of the same model. And because it hallucinates plausible content, it should never be used to claim what a hidden area actually contained, for instance in evidence or documentation.

Running LaMa is inexpensive. The checkpoint is small, inference is one network pass, and IOPaint runs it on a CPU, though a GPU is faster. Training a model from scratch is another matter, as Big LaMa's roughly 1,900 GPU-hours show, and the Places-Challenge training images alone take about 476 GB. The official repository provides pretrained models, Docker scripts for CPU and CUDA, and a mask generator for evaluation.

04Urugero rwakazi

Kuraho ikintu kumafoto hamwe na LaMa

  1. Tora ifoto hamwe nikintu udashaka kuruhande rwinyuma imwe, nkicyapa kumurima wibyatsi.

  2. Kora mask ingana nifoto, ushushanya ikintu nu ntera ntoya izengurutse umweru nibindi byose birabura.

  3. Shyiramo IOPaint hamwe na pipine hanyuma utangire hamwe na moderi ya LaMa kuri CPU, hanyuma ufungure page yaho ikorera.

  4. Fungura ifoto, ohanagura hejuru yikintu, hanyuma ukore ibyuzuye; ubundi shyira ishusho.png nishusho_mask001.png mububiko hanyuma ukore inyandiko yemewe yo guhanura hamwe na big-lama.

  5. Kwegera ahantu huzuye kumurongo wuzuye hanyuma ushakishe ibicucu, inshuro nyinshi cyangwa imirongo igororotse.

  6. Subiramo hamwe nifoto aho ikintu cyicaye imbere yigitereko cyateganijwe, hanyuma ugereranye ubwiza bwombi bwuzuye.

Ibyo byerekana

Ifoto isukuye no kugereranya mu buryo butaziguye ikibazo cyoroshye kandi gikomeye. Amavu n'amavuko agomba kuzuza neza, mugihe tile yerekana ishobora kwerekana imirongo igoramye cyangwa idahuye, uburyo bwo kunanirwa abanditsi b'impapuro basobanura.

Gushyira mu bikorwa Isi

Kurandura abantu batifuzwa, ibintu, inenge cyangwa ibimenyetso byamazi muri IOPaint, igikoresho cya Apache-2.0 gifungura isoko ya Sanster, utegura lama-isukura, ikoresha LaMa mugukuraho kandi ikora kuri CPU, NVIDIA GPU cyangwa Apple Silicon.

Batch ushushanya ububiko bwamafoto hamwe numujyanama wemewe / ububiko bwa lama ubika buri mask nkishusho1_mask001.png kuruhande rwishusho1.png hanyuma ukore bin / guhanura.py hamwe na bariyeri nini ya lama.

Kunoza ibisubizo bihanitse cyane hamwe nuburyo bwo kunonosora uburyo bwa Geomagical Labs, butunganya LaMa imbere yimbere mugihe cyagenwe kandi igashoboka muri code yemewe hamwe na refine = Nukuri.

Gukoresha LaMa ku bikoresho bya Apple ukoresheje CoreMLaMa, umuganda uhinduranya imiterere ya Core ML ya Apple yanditse kuri README yububiko.

Ibipimo byerekana uburyo bushya bwo gushushanya burwanya LaMa ahantu hamwe na CelebA-HQ hamwe na FID na LPIPS, imibare namakuru yakoreshejwe mu mpapuro zumwimerere.

Ejo hazaza h'icyemezo cya LaMa-Igishushanyo gikomeye

The LaMa authors left one question explicitly open in their 2022 paper: whether Fourier convolutions can handle deformed periodic signals, such as patterns seen in strong perspective, where the model usually struggles. High-resolution coherence was a second thread, addressed in 2022 by feature refinement at inference time, which trades extra computation per image for better structure without retraining. The broader direction of inpainting has since moved toward diffusion models that can add content guided by text, and tools such as IOPaint now pair LaMa for removal with diffusion models such as PowerPaint, BrushNet and SDXL inpainting for replacement and outpainting. LaMa's lasting role is as a light, fast removal model and a standard baseline on Places and CelebA-HQ. Open problems for the field include filling holes that hide complex objects or text, measuring realism in ways that match human judgment better than FID, and labeling edited images so that plausible fills are not mistaken for real content.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Umuvuduko n'igipimo. AI igaragara irashobora gukora igenzura, gutahura, no gutondekanya imirimo kurwego.
  • Kubaka amahitamo. Amakipe arema arashobora prototype ibitekerezo byihuse hamwe nintoki nkeya.
  • Itsinda hamwe nakazi. Ibikorwa birashobora gukoresha amashusho nibimenyetso bya videwo byari bigoye gutunganya.

Witondere

  • Uburenganzira bwishusho hamwe no kwemererwa birashobora guhinduka ibyago byemewe n'amategeko niba ibimenyetso bidasobanutse.
  • Imikorere yicyitegererezo irashobora gutandukana kumurika, demografiya, nibidukikije.
  • Ibyiza byibinyoma birashobora kutamenyekana keretse niba ibyiringiro byateganijwe bikurikiranwa.

Intambwe ikurikira

  1. Sobanura ibipimo byo kwemererwa kugiciro, kwibutsa, nibiciro byamakosa.
  2. Gerageza hamwe namakuru ajyanye nuburyo nyabwo bwo gukora.
  3. Ongeraho isubiramo ryabantu kubwizere buke cyangwa guhanura cyane.
  4. Kurikirana icyitegererezo cya drift hanyuma uhindurwe nyuma ya kamera cyangwa dataset ihinduka.

Inkomoko no gusoma

  1. arXiv (WACV 2022)Gukemura-gukomera Mask nini yo gushushanya hamwe na Fourier Convolutions
  2. GitHubumujyanama / lama: LaMa Igishushanyo
  3. NeurIPS 2020Kwihuta Byihuse
  4. arXiv (CVPR 2022 Workshop)Ikiranga kunonosora kunoza imyanzuro ihanitse yo gushushanya
  5. GitHubSanster/IOPaint

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Fata LaMa Icyemezo-Ikibazo Cyimbitse

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

Niki LaMa Icyemezo-Igishushanyo gikomeye?

LaMa (Inkingi nini ya Mask) ni ishusho yerekana ishusho yakozwe nabashakashatsi ba Samsung AI, yerekanwe kuri WACV 2022, yuzuza ibyobo binini mumafoto ukoresheje ibyemezo bya Fourier kugirango ubone ishusho yose icyarimwe. Yahuguwe gusa kuri 256 kubihingwa 256, ikora kumashusho manini cyane, kandi ubunini bwayo buto hamwe na code ifunguye byatumye moteri isanzwe ikuraho ibintu mubikoresho byinshi byubusa.

Nibihe byasinywe byubaka biha LaMa isi yose ibona ishusho?

LaMa ikoresha Byihuta Byihuta (FFCs), ikora igice murwego rwumurongo kugirango itange ikibanza cyakira isi yose murwego rumwe.

Kuki LaMa ikora maska manini kurenza moderi nyinshi zo gushushanya?

Kuberako FFC ibona ishusho yose icyarimwe, LaMa irashobora guhuriza hamwe gukomeza imiterere na geometrie ndetse no mubice byinshi byabuze.

Ni iyihe myitwarire yo gukemura raporo ya LaMa?

Abanditsi bavuga ko LaMa rusange kugeza ku myanzuro irenze-amahugurwa mu bushakashatsi bwabo; imikorere iracyashingira kumashusho na mask.

Ni ubuhe bwoko bw'imiterere LaMa ikora neza cyane bitewe no gutunganya inshuro nyinshi?

Imiyoboro ya Frequency-domaine ihuye neza nuburyo bwigihe, bityo LaMa ikomeza isubiramo imyenda nkurukuta rwamatafari hasi.

Ugereranije na sisitemu nini yo gukwirakwiza ishingiye ku gushushanya, ni gute LaMa ihagaze?

LaMa nuburyo bwo gushushanya bwo gushushanya bwubatswe hafi ya FFC kandi bukoreshwa kenshi aho kuzuza neza ari ingirakamaro; umuvuduko no kwibuka biterwa no gushyira mubikorwa hamwe nibikoresho.