Usanifu wa Picha ya SPADE Semantiki

Semantic image synthesis is the task of producing a realistic photograph from a semantic segmentation map. A segmentation map is an image in which every pixel holds a class label rather than a color: this region is sky, that region is mountain, the strip at the bottom is water. It is the reverse of semantic segmentation, where a network labels each pixel of a photo. The appeal is control. A user decides exactly what goes where, and the model fills in texture, lighting, reflections and detail. SPADE, introduced by Taesung Park, Ming-Yu Liu, Ting-Chun Wang and Jun-Yan Zhu in "Semantic Image Synthesis with Spatially-Adaptive Normalization" (arXiv March 2019, oral presentation at CVPR 2019), made this task work much better by changing one layer.

The problem SPADE fixed is subtle. Earlier generators such as pix2pixHD fed the segmentation map into the first layer of a convolutional network and then passed it through stacks of convolution, normalization and activation layers. Normalization layers, such as batch or instance normalization, rescale each channel's activations to zero mean and unit variance so training stays stable. The paper points out that a segmentation map consists mostly of large uniform regions. If a whole region carries one label, convolution produces a uniform output there, and instance normalization turns that uniform output into zeros, regardless of which label it was. The layout information is, in the authors' words, washed away. Their Figure 3 shows pix2pixHD producing identical images from two different single-label maps.

SPADE's mechanism works in four steps. First, activations in a layer are normalized per channel, like batch normalization, but without a learned scale and shift. Second, the segmentation map is resized to that layer's resolution. Third, a small two-layer convolutional network reads the resized map and outputs two tensors, gamma and beta, that have the same height and width as the activations, so they vary from pixel to pixel. Fourth, the normalized activations are multiplied by gamma and added to beta. Because the scale and shift depend on the label at each location, the network re-injects layout information at every layer instead of only at the input. The generator is a stack of residual blocks with SPADE layers and upsampling, starting from a small random vector, and the encoder half used in earlier generators is removed, making the network lighter.

Training uses a generative adversarial network (GAN): a generator makes images and a discriminator tries to tell them from real photos. SPADE reused the multi-scale discriminator and loss terms of pix2pixHD, swapping its least-squares adversarial loss for a hinge loss, and kept a perceptual loss that compares deep features of generated and real images. An optional image encoder turns a reference photo into a style vector, forming a variational autoencoder (VAE) with the generator. At test time that encoder lets a user pick a style image, such as a sunset photo, while the segmentation map controls the layout. The experiments ran on an Nvidia DGX-1 with eight 32 GB V100 GPUs.

The results were a clear step forward. On COCO-Stuff, with 118,000 training images and 182 classes, SPADE reached a mean intersection over union (mIoU, a measure of how well a segmentation network can recover the input layout from the generated image) of 37.4, against 23.7 for the cascaded refinement network (CRN) and 14.6 for pix2pixHD, and a Frechet Inception Distance (FID, lower means more realistic) of 22.6, against 70.4 and 111.5. On ADE20K, with 150 classes, SPADE scored 38.5 mIoU and 33.9 FID. On Cityscapes street scenes, the semi-parametric SIMS method, which stitches real image patches, had a lower FID of 49.7 against SPADE's 71.8, but weaker layout alignment, since copied patches may not match the requested segments.

NVIDIA turned the research into GauGAN, announced on March 18, 2019 and named after the painter Paul Gauguin. NVIDIA said its discriminator was trained on a million real images and that the model was built in PyTorch. Bryan Catanzaro, NVIDIA's vice president of applied deep learning research, described it as a "smart paintbrush". GauGAN won two awards at SIGGRAPH 2019 Real-Time Live. A later version, GauGAN2, added the ability to generate a scene from a short text phrase and then edit it with the segmentation brush, and the technology was packaged as NVIDIA Canvas, a desktop app for RTX GPUs. NVIDIA also released the SPADE source code for non-commercial use.

Compared with other approaches, SPADE sits between earlier image-to-image translation and today's diffusion models. Against pix2pixHD and CRN, it gave better realism and layout fidelity with fewer parameters. Against SIMS, it synthesized new content instead of retrieving patches. Later GAN work improved on it: OASIS (Sushko and colleagues, ICLR 2021) replaced the discriminator with a segmentation network and dropped the perceptual loss, reporting average gains of 6 FID and 5 mIoU points over the prior state of the art. Since 2023, diffusion-based methods such as ControlNet have added segmentation-map control to large text-to-image models trained on billions of images, giving more diverse content at the cost of slower generation.

SPADE has clear limits. It needs paired data, images with pixel-accurate segmentation maps, which are expensive to label by hand; the Flickr landscape experiment sidestepped this by labeling photos automatically with a segmentation network, so label errors pass into training. Output quality falls on rare classes and complex scenes with many small objects, and COCO-Stuff images were still visibly imperfect. As a GAN, it can suffer from training instability and limited diversity for a given map unless the style encoder or noise input is used. Its vocabulary is fixed to the training classes: a map can only contain labels the model saw. Resolution was modest by current standards: the paper trained at 256 by 256 pixels, or 512 by 256 for Cityscapes.