概述
它將地圖輸入到每個正規化層中,因此佈局資訊不會被沖刷掉,它為 NVIDIA 的 GauGAN 繪圖演示提供了動力,後來也應用於 NVIDIA Canvas 應用程式。
重點摘要
- SPADE 是一種歸一化層,其縮放與位移會根據輸入的分割圖按像素變化,因此佈局資訊可以傳遞至每一個生成器層。
- 2019 年的 SPADE 論文顯示,普通的歸一化可能會抹除語義資訊,因為分割圖中包含大面積的均勻區域。
- 在 COCO-Stuff 測試集上,SPADE 達到了 37.4 mIoU 和 22.6 FID,而早期的 pix2pixHD 模型則為 14.6 mIoU 和 111.5 FID。
- NVIDIA 於 2019 年 3 月發布的 GauGAN 演示使用 SPADE 將繪製的標籤圖轉換為風景,其技術後來應用於 NVIDIA Canvas。
- OASIS 和 ControlNet 等較新的方法改進了佈局控制生成,其中 ControlNet 在 2023 年為擴散模型增加了分割控制。
深入探討
Semantic image synthesis is the task of producing a realistic photograph from a semantic segmentation map. A segmentation map is an image in which every pixel holds a class label rather than a color: this region is sky, that region is mountain, the strip at the bottom is water. It is the reverse of semantic segmentation, where a network labels each pixel of a photo. The appeal is control. A user decides exactly what goes where, and the model fills in texture, lighting, reflections and detail. SPADE, introduced by Taesung Park, Ming-Yu Liu, Ting-Chun Wang and Jun-Yan Zhu in "Semantic Image Synthesis with Spatially-Adaptive Normalization" (arXiv March 2019, oral presentation at CVPR 2019), made this task work much better by changing one layer.
The problem SPADE fixed is subtle. Earlier generators such as pix2pixHD fed the segmentation map into the first layer of a convolutional network and then passed it through stacks of convolution, normalization and activation layers. Normalization layers, such as batch or instance normalization, rescale each channel's activations to zero mean and unit variance so training stays stable. The paper points out that a segmentation map consists mostly of large uniform regions. If a whole region carries one label, convolution produces a uniform output there, and instance normalization turns that uniform output into zeros, regardless of which label it was. The layout information is, in the authors' words, washed away. Their Figure 3 shows pix2pixHD producing identical images from two different single-label maps.
SPADE's mechanism works in four steps. First, activations in a layer are normalized per channel, like batch normalization, but without a learned scale and shift. Second, the segmentation map is resized to that layer's resolution. Third, a small two-layer convolutional network reads the resized map and outputs two tensors, gamma and beta, that have the same height and width as the activations, so they vary from pixel to pixel. Fourth, the normalized activations are multiplied by gamma and added to beta. Because the scale and shift depend on the label at each location, the network re-injects layout information at every layer instead of only at the input. The generator is a stack of residual blocks with SPADE layers and upsampling, starting from a small random vector, and the encoder half used in earlier generators is removed, making the network lighter.
Training uses a generative adversarial network (GAN): a generator makes images and a discriminator tries to tell them from real photos. SPADE reused the multi-scale discriminator and loss terms of pix2pixHD, swapping its least-squares adversarial loss for a hinge loss, and kept a perceptual loss that compares deep features of generated and real images. An optional image encoder turns a reference photo into a style vector, forming a variational autoencoder (VAE) with the generator. At test time that encoder lets a user pick a style image, such as a sunset photo, while the segmentation map controls the layout. The experiments ran on an Nvidia DGX-1 with eight 32 GB V100 GPUs.
The results were a clear step forward. On COCO-Stuff, with 118,000 training images and 182 classes, SPADE reached a mean intersection over union (mIoU, a measure of how well a segmentation network can recover the input layout from the generated image) of 37.4, against 23.7 for the cascaded refinement network (CRN) and 14.6 for pix2pixHD, and a Frechet Inception Distance (FID, lower means more realistic) of 22.6, against 70.4 and 111.5. On ADE20K, with 150 classes, SPADE scored 38.5 mIoU and 33.9 FID. On Cityscapes street scenes, the semi-parametric SIMS method, which stitches real image patches, had a lower FID of 49.7 against SPADE's 71.8, but weaker layout alignment, since copied patches may not match the requested segments.
NVIDIA turned the research into GauGAN, announced on March 18, 2019 and named after the painter Paul Gauguin. NVIDIA said its discriminator was trained on a million real images and that the model was built in PyTorch. Bryan Catanzaro, NVIDIA's vice president of applied deep learning research, described it as a "smart paintbrush". GauGAN won two awards at SIGGRAPH 2019 Real-Time Live. A later version, GauGAN2, added the ability to generate a scene from a short text phrase and then edit it with the segmentation brush, and the technology was packaged as NVIDIA Canvas, a desktop app for RTX GPUs. NVIDIA also released the SPADE source code for non-commercial use.
Compared with other approaches, SPADE sits between earlier image-to-image translation and today's diffusion models. Against pix2pixHD and CRN, it gave better realism and layout fidelity with fewer parameters. Against SIMS, it synthesized new content instead of retrieving patches. Later GAN work improved on it: OASIS (Sushko and colleagues, ICLR 2021) replaced the discriminator with a segmentation network and dropped the perceptual loss, reporting average gains of 6 FID and 5 mIoU points over the prior state of the art. Since 2023, diffusion-based methods such as ControlNet have added segmentation-map control to large text-to-image models trained on billions of images, giving more diverse content at the cost of slower generation.
SPADE has clear limits. It needs paired data, images with pixel-accurate segmentation maps, which are expensive to label by hand; the Flickr landscape experiment sidestepped this by labeling photos automatically with a segmentation network, so label errors pass into training. Output quality falls on rare classes and complex scenes with many small objects, and COCO-Stuff images were still visibly imperfect. As a GAN, it can suffer from training instability and limited diversity for a given map unless the style encoder or noise input is used. Its vocabulary is fixed to the training classes: a map can only contain labels the model saw. Resolution was modest by current standards: the paper trained at 256 by 256 pixels, or 512 by 256 for Cityscapes.
04工作範例
使用預訓練的 SPADE 模型將三區域草圖轉換為風景
下載官方的 NVlabs SPADE 程式碼和預訓練檢查點(例如 ADE20K 或 COCO-Stuff 模型),並遵循儲存庫中的說明。
建立一張 256 x 256 的標籤圖:使用資料集的類別 ID,將頂部三分之一填滿「天空」類別,中間填滿「山脈」類別,底部填滿「水」類別。
將其儲存為單通道 PNG,其中每個像素值為類別 ID,而非顏色。
對該標籤圖執行測試腳本,以從隨機風格向量生成圖像。
將底部區域的類別從「水」更改為「草地」並再次執行,同時保持其他所有內容不變。
如果模型包含風格編碼器,請提供一張日落照片作為風格圖像,並生成第三個版本。
它顯示了什麼
這三張圖像共享相同的佈局,但在內容或風格上有所不同:湖泊場景、草地場景和日落版本。透過在每個輸出上執行分割網路並將其標籤與輸入地圖進行比較,來檢查佈局的保真度。
現實世界的實施
NVIDIA 的 GauGAN 演示於 2019 年 3 月的 GPU 技術大會上展示,讓參觀者繪製標記為「海」、「沙」或「雪」的區域,並觀看 SPADE 模型在 TITAN RTX GPU 上渲染出相匹配的風景。
GauGAN 在 SIGGRAPH 2019 Real-Time Live 競賽中贏得了「最佳展示獎」和「觀眾選擇獎」,NVIDIA 和加州大學柏克萊分校的研究人員在該競賽中於配備手寫顯示器的 RTX Studio 筆記型電腦上進行了現場演示。
NVIDIA Canvas 是一款在 NVIDIA RTX GPU 上執行的桌面應用程式,它將 GauGAN 技術帶給藝術家,讓他們能繪製如「草地」或「雲朵」等標籤區域,並能與 Adobe Photoshop 等工具協同工作。
在 2019 年的論文中,一個在 41,000 張 Flickr 風景照片上訓練的 SPADE 模型(由 DeepLabV2 分割網路自動標註)成功從使用者編輯的佈局中生成了新的風景。
建築師、城市規劃師、景觀設計師和遊戲開發者被 NVIDIA 列為潛在使用者,他們可以利用此技術從粗略的佈局中快速製作合成場景原型。
SPADE語意影像合成的未來
Since SPADE, research on layout-controlled image generation has moved in two directions. GAN-based methods kept refining the training signal, as in OASIS (ICLR 2021), which treats the discriminator as a segmentation network and reports average gains of 6 FID and 5 mIoU points without perceptual losses. Diffusion-based methods now dominate new work: ControlNet (Zhang, Rao and Agrawala, 2023) attaches spatial conditions such as segmentation maps, edges, depth and pose to a frozen Stable Diffusion model through zero-initialized convolutions, and reports robust training on datasets from under 50,000 to over a million images. NVIDIA's own product path also changed. GauGAN2 combined segmentation painting with text prompts, and NVIDIA Canvas brought the approach to artists as a free desktop app. Open problems include generating many small objects correctly, keeping open vocabularies while preserving the precise layout control that segmentation maps offer, speeding up diffusion models enough for the real-time interaction GauGAN demonstrated, and reducing dependence on costly pixel-level labels. The SPADE layer itself, spatially varying modulation of normalized features, remains a reusable idea that appears in later conditional generators.
付諸實踐
為什麼這很重要
- 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
- 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
- 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。
留意
- 如果出處不明,肖像權和同意可能會成為法律風險。
- 模型表現可能因光照、人口統計和環境的不同而有所不同。
- 除非監控置信閾值,否則誤報可能會被忽略。
後續步驟
- 定義精確度、召回率和錯誤成本的接受標準。
- 使用符合實際生產條件的數據進行測試。
- 為低置信度或高影響力的預測添加人工審核。
- 追蹤模型漂移並在相機或資料集變更後重新驗證。
資料來源與延伸閱讀
- arXiv (CVPR 2019)基於空間自適應歸一化的語義圖像合成
- NVIDIA Blog天才之作:GauGAN 將塗鴉轉變為令人驚嘆的逼真風景
- NVIDIA Blog什麼是 GauGAN?AI 如何將文字和圖片轉化為藝術
- NVIDIA Technical BlogGauGAN 在 SIGGRAPH 2019 Real-Time Live 競賽中贏得大獎
- arXiv (ICLR 2021)語義圖像合成僅需對抗性監督
- arXiv為文字轉圖像擴散模型增加條件控制
不斷探索
常見問題
什麼是 SPADE 語意影像合成?
SPADE 是「空間自適應歸一化」(spatially-adaptive normalization) 的縮寫,是來自 NVIDIA 研究人員 (CVPR 2019) 的一種神經網路層,它能將語義地圖(一種將每個區域標記為天空、樹木或水的圖片)轉換為逼真的圖像。它將地圖輸入到每個歸一化層中,以確保佈局資訊不會被沖淡,這項技術為 NVIDIA 的 GauGAN 繪畫演示以及後來的 NVIDIA Canvas 應用程式提供了動力。
SPADE 使用什麼輸入來產生影像?
SPADE 從語義分割圖中合成真實感影像,其中每個像素都帶有天空或草地等類別標籤。
SPADE 修復了早期標準化的什麼問題?
傳統的標準化往往會擦除空間語義訊息,因此 SPADE 在整個網路中重新註入佈局。
哪個著名的互動式應用程式是基於 SPADE 建構的?
NVIDIA 的 GauGAN(後來的 NVIDIA Canvas)可讓使用者繪製標籤地圖,然後 SPADE 將其轉換為逼真的場景。
SPADE 中的「SP」代表什麼?
SPADE 代表空間自適應(去)歸一化,指的是其位置相關的調製。
為什麼 SPADE 生成的影像在訓練語意分割系統時很有用?
本指南列出了產生具有已知像素標籤的各種合成影像,作為開發分割模型的實用方法。