Visuele AI-GIDS

SPADE Semantische beeldsynthese

SPADE, short for spatially-adaptive normalization, is a neural network layer from NVIDIA researchers (CVPR 2019) that turns a semantic map, a picture where each region is labeled sky, tree or water, into a photorealistic image.

Luister naar deze gidsAI-verteld · ~8 min

Op deze pagina9 minuten lezen
  1. Overzicht
  2. Belangrijkste inzichten
  3. Diepe duik
  4. Turning a three-region sketch into a landscape with a pretrained SPADE model
  5. Implementatie in de echte wereld
  6. De toekomst van SPADE semantische beeldsynthese
  7. Het in de praktijk brengen
  8. Bronnen en verdere literatuur
  9. Blijf verkennen
  10. Veelgestelde vragen

Overzicht

It feeds the map into every normalization layer so layout information is not washed out, and it powered NVIDIA's GauGAN painting demo and later the NVIDIA Canvas app.

Belangrijkste inzichten

  1. SPADE is a normalization layer whose scale and shift vary per pixel according to the input segmentation map, so layout information reaches every generator layer.
  2. The 2019 SPADE paper showed that ordinary normalization can erase semantic information because segmentation maps contain large uniform regions.
  3. On COCO-Stuff, SPADE reached 37.4 mIoU and 22.6 FID, against 14.6 mIoU and 111.5 FID for the earlier pix2pixHD model.
  4. NVIDIA's GauGAN demo, announced in March 2019, used SPADE to turn painted label maps into landscapes, and its technology later shipped in NVIDIA Canvas.
  5. Newer methods such as OASIS and ControlNet improved layout-controlled generation, with ControlNet adding segmentation control to diffusion models in 2023.

Diepe duik

Semantic image synthesis is the task of producing a realistic photograph from a semantic segmentation map. A segmentation map is an image in which every pixel holds a class label rather than a color: this region is sky, that region is mountain, the strip at the bottom is water. It is the reverse of semantic segmentation, where a network labels each pixel of a photo. The appeal is control. A user decides exactly what goes where, and the model fills in texture, lighting, reflections and detail. SPADE, introduced by Taesung Park, Ming-Yu Liu, Ting-Chun Wang and Jun-Yan Zhu in "Semantic Image Synthesis with Spatially-Adaptive Normalization" (arXiv March 2019, oral presentation at CVPR 2019), made this task work much better by changing one layer.

The problem SPADE fixed is subtle. Earlier generators such as pix2pixHD fed the segmentation map into the first layer of a convolutional network and then passed it through stacks of convolution, normalization and activation layers. Normalization layers, such as batch or instance normalization, rescale each channel's activations to zero mean and unit variance so training stays stable. The paper points out that a segmentation map consists mostly of large uniform regions. If a whole region carries one label, convolution produces a uniform output there, and instance normalization turns that uniform output into zeros, regardless of which label it was. The layout information is, in the authors' words, washed away. Their Figure 3 shows pix2pixHD producing identical images from two different single-label maps.

SPADE's mechanism works in four steps. First, activations in a layer are normalized per channel, like batch normalization, but without a learned scale and shift. Second, the segmentation map is resized to that layer's resolution. Third, a small two-layer convolutional network reads the resized map and outputs two tensors, gamma and beta, that have the same height and width as the activations, so they vary from pixel to pixel. Fourth, the normalized activations are multiplied by gamma and added to beta. Because the scale and shift depend on the label at each location, the network re-injects layout information at every layer instead of only at the input. The generator is a stack of residual blocks with SPADE layers and upsampling, starting from a small random vector, and the encoder half used in earlier generators is removed, making the network lighter.

Training uses a generative adversarial network (GAN): a generator makes images and a discriminator tries to tell them from real photos. SPADE reused the multi-scale discriminator and loss terms of pix2pixHD, swapping its least-squares adversarial loss for a hinge loss, and kept a perceptual loss that compares deep features of generated and real images. An optional image encoder turns a reference photo into a style vector, forming a variational autoencoder (VAE) with the generator. At test time that encoder lets a user pick a style image, such as a sunset photo, while the segmentation map controls the layout. The experiments ran on an Nvidia DGX-1 with eight 32 GB V100 GPUs.

The results were a clear step forward. On COCO-Stuff, with 118,000 training images and 182 classes, SPADE reached a mean intersection over union (mIoU, a measure of how well a segmentation network can recover the input layout from the generated image) of 37.4, against 23.7 for the cascaded refinement network (CRN) and 14.6 for pix2pixHD, and a Frechet Inception Distance (FID, lower means more realistic) of 22.6, against 70.4 and 111.5. On ADE20K, with 150 classes, SPADE scored 38.5 mIoU and 33.9 FID. On Cityscapes street scenes, the semi-parametric SIMS method, which stitches real image patches, had a lower FID of 49.7 against SPADE's 71.8, but weaker layout alignment, since copied patches may not match the requested segments.

NVIDIA turned the research into GauGAN, announced on March 18, 2019 and named after the painter Paul Gauguin. NVIDIA said its discriminator was trained on a million real images and that the model was built in PyTorch. Bryan Catanzaro, NVIDIA's vice president of applied deep learning research, described it as a "smart paintbrush". GauGAN won two awards at SIGGRAPH 2019 Real-Time Live. A later version, GauGAN2, added the ability to generate a scene from a short text phrase and then edit it with the segmentation brush, and the technology was packaged as NVIDIA Canvas, a desktop app for RTX GPUs. NVIDIA also released the SPADE source code for non-commercial use.

Compared with other approaches, SPADE sits between earlier image-to-image translation and today's diffusion models. Against pix2pixHD and CRN, it gave better realism and layout fidelity with fewer parameters. Against SIMS, it synthesized new content instead of retrieving patches. Later GAN work improved on it: OASIS (Sushko and colleagues, ICLR 2021) replaced the discriminator with a segmentation network and dropped the perceptual loss, reporting average gains of 6 FID and 5 mIoU points over the prior state of the art. Since 2023, diffusion-based methods such as ControlNet have added segmentation-map control to large text-to-image models trained on billions of images, giving more diverse content at the cost of slower generation.

SPADE has clear limits. It needs paired data, images with pixel-accurate segmentation maps, which are expensive to label by hand; the Flickr landscape experiment sidestepped this by labeling photos automatically with a segmentation network, so label errors pass into training. Output quality falls on rare classes and complex scenes with many small objects, and COCO-Stuff images were still visibly imperfect. As a GAN, it can suffer from training instability and limited diversity for a given map unless the style encoder or noise input is used. Its vocabulary is fixed to the training classes: a map can only contain labels the model saw. Resolution was modest by current standards: the paper trained at 256 by 256 pixels, or 512 by 256 for Cityscapes.

04Uitgewerkt voorbeeld

Turning a three-region sketch into a landscape with a pretrained SPADE model

  1. Download the official NVlabs SPADE code and a pretrained checkpoint, such as the ADE20K or COCO-Stuff model, following the repository instructions.

  2. Create a 256 by 256 label map: fill the top third with the sky class, the middle with the mountain class and the bottom with the water class, using the dataset's class IDs.

  3. Save it as a single-channel PNG in which each pixel value is a class ID, not a color.

  4. Run the test script on the label map to generate an image from a random style vector.

  5. Change the bottom region's class from water to grass and run again, keeping everything else fixed.

  6. If the model includes the style encoder, supply a sunset photo as the style image and generate a third version.

Wat het laat zien

Three images share the same layout but differ in content or style: a lake scene, a meadow scene and a sunset version. Check layout fidelity by running a segmentation network on each output and comparing its labels to the input map.

Implementatie in de echte wereld

NVIDIA's GauGAN demo, shown at the GPU Technology Conference in March 2019, let visitors paint regions labeled sea, sand or snow and watch a SPADE model render a matching landscape on a TITAN RTX GPU.

GauGAN won Best of Show and Audience Choice at the SIGGRAPH 2019 Real-Time Live competition, where NVIDIA and UC Berkeley researchers demonstrated it live on an RTX Studio laptop with a pen display.

NVIDIA Canvas, a desktop app that runs on NVIDIA RTX GPUs, brings GauGAN technology to artists who paint labeled regions such as grass or clouds and works alongside tools like Adobe Photoshop.

In the 2019 paper, a SPADE model trained on 41,000 Flickr landscape photos, labeled automatically by a DeepLabV2 segmentation network, produced new landscapes from user-edited layouts.

Architects, urban planners, landscape designers and game developers were named by NVIDIA as potential users for quickly prototyping synthetic scenes from rough layouts.

De toekomst van SPADE semantische beeldsynthese

Since SPADE, research on layout-controlled image generation has moved in two directions. GAN-based methods kept refining the training signal, as in OASIS (ICLR 2021), which treats the discriminator as a segmentation network and reports average gains of 6 FID and 5 mIoU points without perceptual losses. Diffusion-based methods now dominate new work: ControlNet (Zhang, Rao and Agrawala, 2023) attaches spatial conditions such as segmentation maps, edges, depth and pose to a frozen Stable Diffusion model through zero-initialized convolutions, and reports robust training on datasets from under 50,000 to over a million images. NVIDIA's own product path also changed. GauGAN2 combined segmentation painting with text prompts, and NVIDIA Canvas brought the approach to artists as a free desktop app. Open problems include generating many small objects correctly, keeping open vocabularies while preserving the precise layout control that segmentation maps offer, speeding up diffusion models enough for the real-time interaction GauGAN demonstrated, and reducing dependence on costly pixel-level labels. The SPADE layer itself, spatially varying modulation of normalized features, remains a reusable idea that appears in later conditional generators.

Het in de praktijk brengen

Waarom het ertoe doet

  • Snelheid en schaal. Visuele AI kan inspectie-, detectie- en taggingtaken op schaal automatiseren.
  • Bouwkeuzes. Creatieve teams kunnen concepten sneller prototypen met minder handmatige revisies.
  • Team en workflow. Bij bewerkingen kan gebruik worden gemaakt van beeld- en videosignalen die voorheen moeilijk te verwerken waren.

Pas op voor

  • Beeldrechten en toestemming kunnen juridische risico's worden als de herkomst onduidelijk is.
  • De prestaties van modellen kunnen variëren afhankelijk van de belichting, demografische gegevens en omgevingen.
  • Valse positieve resultaten kunnen onopgemerkt blijven, tenzij de vertrouwensdrempels worden gecontroleerd.

Volgende stappen

  1. Definieer acceptatiecriteria voor precisie-, terugroep- en foutkosten.
  2. Test met gegevens die overeenkomen met echte productieomstandigheden.
  3. Voeg menselijke beoordeling toe voor voorspellingen met weinig vertrouwen of hoge impact.
  4. Volg modelafwijkingen en valideer opnieuw na wijzigingen in de camera of dataset.

Bronnen en verdere literatuur

  1. arXiv (CVPR 2019)Semantic Image Synthesis with Spatially-Adaptive Normalization
  2. NVIDIA BlogStroke of Genius: GauGAN Turns Doodles into Stunning, Photorealistic Landscapes
  3. NVIDIA BlogWhat Is GauGAN? How AI Turns Words and Pictures Into Art
  4. NVIDIA Technical BlogGauGAN Wins Major Awards at SIGGRAPH 2019's Real Time Live Competition
  5. arXiv (ICLR 2021)You Only Need Adversarial Supervision for Semantic Image Synthesis
  6. arXivAdding Conditional Control to Text-to-Image Diffusion Models

Blijf verkennen

Gratis nieuwsbrief

Ontvang de dagelijkse AI-briefing

Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.

Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.

Test jezelf

Take the SPADE Semantic Image Synthesis quiz

Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.

Quiz starten

Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie

Veelgestelde vragen

Wat is SPADE semantische beeldsynthese?

SPADE, short for spatially-adaptive normalization, is a neural network layer from NVIDIA researchers (CVPR 2019) that turns a semantic map, a picture where each region is labeled sky, tree or water, into a photorealistic image. It feeds the map into every normalization layer so layout information is not washed out, and it powered NVIDIA's GauGAN painting demo and later the NVIDIA Canvas app.

Welke input gebruikt SPADE om een ​​afbeelding te genereren?

SPADE synthetiseert fotorealistische beelden uit semantische segmentatiekaarten waarbij elke pixel een categorielabel draagt, zoals lucht of gras.

Welk probleem met eerdere normalisatie heeft SPADE opgelost?

Conventionele normalisatie had de neiging de ruimtelijke semantische informatie te wissen, dus SPADE injecteert de lay-out opnieuw door het hele netwerk.

Welke bekende interactieve app is gebouwd op SPADE?

Met NVIDIA's GauGAN, later NVIDIA Canvas, kunnen gebruikers labelkaarten schilderen die SPADE in fotorealistische scènes verandert.

Waar staat de 'SP' in SPADE voor?

SPADE staat voor Spatially-Adaptive (De)normalization, verwijzend naar de locatieafhankelijke modulatie.

Why can SPADE-generated images be useful when training semantic-segmentation systems?

The guide lists generating varied synthetic images with known pixel labels as a practical way to develop segmentation models.