視覺人工智慧指南

VQGAN 和 Codebook 影像合成

VQGAN is a neural network that compresses an image into a small grid of entries from a learned codebook, a dictionary of reusable visual parts, and decodes that grid back into a sharp picture.

聆聽此指南AI 語音敘述 · ~9 min

本頁閱讀時間 10 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. Trace one image through a VQGAN pipeline
  5. 現實世界的實施
  6. VQGAN 和 Codebook 影像合成的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

Introduced in 2020, it made transformer image generation practical at high resolution and its tokenizer idea still underpins many text-to-image and multimodal models.

重點摘要

  1. VQGAN turns an image into a small grid of indices from a learned codebook, so image generation becomes a token prediction problem.
  2. The 2020 Taming Transformers paper by Esser, Rombach and Ommer introduced VQGAN with a default 16 by 16 token grid for 256 by 256 images.
  3. Perceptual and adversarial losses let VQGAN compress far harder than VQ-VAE while keeping reconstructions sharp.
  4. VQGAN-CLIP, MaskGIT, ViT-VQGAN and Parti all built directly on the VQGAN tokenizer idea between 2021 and 2022.
  5. Codebook collapse, slow sequential sampling and loss of fine detail are the main known weaknesses of codebook image synthesis.

深入探討

VQGAN, short for Vector Quantized Generative Adversarial Network, is an image autoencoder with a twist. An autoencoder is a pair of networks: an encoder that squeezes an image into a compact representation and a decoder that rebuilds the image from it. In VQGAN the compact representation is not a grid of arbitrary numbers but a grid of indices into a codebook, a learned table of vectors that act like a vocabulary of visual parts such as a patch of fur, a stretch of sky or the corner of a window. Once an image is a grid of codebook indices, it looks like a sentence made of tokens, and any model that is good at predicting the next token in text can be trained to generate images.

The method was introduced by Patrick Esser, Robin Rombach and Bjorn Ommer of Heidelberg University in the paper Taming Transformers for High-Resolution Image Synthesis, first posted to arXiv on 17 December 2020 and revised in June 2021. Their motivation was cost. Transformers, the attention-based networks behind modern language models, compare every element of a sequence with every other element, so the cost grows with the square of the sequence length. A 256 by 256 image has 65,536 pixels, far too many to treat as a sequence. Convolutional neural networks, which process images with small sliding filters, are efficient at local texture but weaker at long-range structure. The paper's answer was to let a convolutional VQGAN learn the vocabulary of local parts and let a transformer learn how those parts are arranged.

The mechanism works in two stages. In stage one, the encoder downsamples the image by a factor f; the paper's default is f = 16, so a 256 by 256 image becomes a 16 by 16 grid of feature vectors. Each vector is replaced by its nearest neighbor in the codebook, a step called vector quantization. The default codebook holds 1,024 entries, with larger codebooks of 4,096 to 16,384 entries used for some datasets. The decoder then reconstructs the image from the quantized grid. Because choosing the nearest entry is not differentiable, gradients are copied straight through the quantizer during training, a trick called the straight-through estimator. In stage two, the encoder and codebook are frozen, every training image is converted to its 256 indices, and a transformer is trained to predict each index from the ones before it. To generate, the transformer samples indices one by one and the decoder turns the finished grid into pixels.

What separates VQGAN from its predecessor is the training loss. VQ-VAE, introduced by Aaron van den Oord, Oriol Vinyals and Koray Kavukcuoglu at DeepMind in November 2017, established the codebook idea and showed that discrete latents avoid posterior collapse, a failure where a powerful decoder learns to ignore the latent code. But VQ-VAE was trained mainly to reproduce pixels, which pushes reconstructions toward blur when the code is heavily compressed. VQGAN adds two terms. A perceptual loss compares images in the feature space of a pretrained network rather than pixel by pixel, and a patch-based discriminator, the adversarial half of a generative adversarial network (GAN), penalizes local patches that look fake. The weight on the adversarial term is set automatically from the ratio of gradient sizes at the decoder's last layer. The result is that VQGAN can compress much harder while staying sharp. The paper reports that its ImageNet VQGAN with 16,384 codes on a 16 by 16 grid reaches a reconstruction FID of 4.98, against 32.01 for the DALL-E tokenizer with 8,192 codes on a 32 by 32 grid, while using a sequence one quarter as long. FID, or Frechet Inception Distance, measures how far the statistics of generated images are from real ones; lower is better.

The generation results were strong for their time. With a GPT-2-medium-sized transformer of about 307 million parameters, the paper reports FIDs of 9.6 on the FFHQ face dataset and 10.2 on CelebA-HQ. On class-conditional ImageNet at 256 by 256 pixels it reports an FID near 16 without rejection sampling and 5.20 with it, compared with about 31 for VQ-VAE-2 without rejection, using roughly a tenth of VQ-VAE-2's estimated parameter count. For images larger than the training size, the transformer attends only to a sliding window of nearby tokens, which allowed the authors to produce megapixel images guided by segmentation or depth maps.

VQGAN quickly became a building block. The VQGAN-CLIP method, written up by Katherine Crowson, Stella Biderman and colleagues for ECCV 2022, paired a pretrained VQGAN with OpenAI's CLIP model and optimized the latent grid until CLIP judged the decoded image to match a text prompt, with no task-specific training. Google's ViT-VQGAN (October 2021) replaced the convolutional encoder and decoder with Vision Transformers and improved codebook learning, raising Inception Score on ImageNet from 70.6 for vanilla VQGAN to 175.1 and lowering FID from 17.04 to 4.17. MaskGIT (February 2022) kept VQ tokens but generated them in parallel by iteratively filling in masked positions. Parti (June 2022) scaled an autoregressive Transformer on ViT-VQGAN tokens to 20 billion parameters. The same Heidelberg group's Latent Diffusion paper (December 2021) reused the idea of working in the compressed space of a pretrained autoencoder, but ran a diffusion model there instead of a token predictor.

Compared with alternatives, codebook synthesis has clear trade-offs. Against pixel-space GANs, it separates compression from generation, so the generator can be a large, stable, likelihood-trained transformer rather than an adversarial network that is prone to training collapse. Against pixel-space diffusion, it is far cheaper because the model works on a few hundred tokens rather than tens of thousands of pixels. Against latent diffusion with continuous latents, discrete tokens fit naturally into the same sequence as text tokens, which is why tokenizers of this family appear in models that mix text and images. The cost is that quantization throws information away, so fine detail such as small text or faces in a crowd can be lost before generation even begins.

The main failure modes are well documented. Codebook collapse, where only a small fraction of entries is ever selected, wastes capacity; the Taming Transformers authors note that their ImageNet model effectively used 973 of its 16,384 entries. Autoregressive sampling is slow because a 256-token image needs 256 sequential steps, which motivated MaskGIT's parallel decoding. Raster-order generation, row by row, can produce inconsistent global layouts, and outputs depend heavily on sampling settings such as top-k. Finally, a 16-fold downsampling limits how much detail any downstream model can express.

In practice the requirements are modest by current standards. The paper reports that most of its transformers were sized to train with batch size 2 on a 12 GB GPU, typically using two to four GPUs, and that an updated ImageNet model trained for 45.8 days on a single A100. Training a new VQGAN still needs a large image dataset matched to the target domain, careful tuning of the adversarial weight and a warm-up period before the discriminator switches on. The authors released code and pretrained models, so encoding and decoding with an existing VQGAN needs far less compute than training one.

04工作範例

Trace one image through a VQGAN pipeline

  1. Start with a 256 by 256 RGB photo, which contains 65,536 pixel positions.

  2. Pass it through the encoder with downsampling factor 16 to get a 16 by 16 grid of 256 feature vectors.

  3. For each vector, find the closest of the 1,024 codebook entries and record its index, giving 256 integers between 0 and 1,023.

  4. Feed the 256 indices to the decoder and compare the rebuilt image with the original by eye and with a perceptual metric.

  5. Mask the bottom half of the grid, then let a trained transformer sample the missing 128 indices one by one using top-k sampling.

  6. Decode the completed grid and note how the new bottom half continues the textures and layout of the top half.

它顯示了什麼

The learner ends with an image whose top half reconstructs the input and whose bottom half is generated. A sharp reconstruction confirms the codebook captures the image well, and a coherent completion confirms the transformer learned how codebook parts fit together.

現實世界的實施

The original Taming Transformers paper used a VQGAN plus a GPT-2-style transformer to generate megapixel landscapes from semantic segmentation maps, painting the image patch by patch with a sliding attention window.

VQGAN-CLIP, published by Katherine Crowson and colleagues and accepted at ECCV 2022, steers a pretrained VQGAN with the CLIP text-image model to create and edit images from free-text prompts without any extra training.

Google's MaskGIT (2022) predicts masked VQ image tokens in parallel instead of one at a time, which the authors report speeds up decoding by up to 64 times compared with autoregressive generation.

Google's Parti text-to-image model (2022) used the improved ViT-VQGAN tokenizer and a Transformer of up to 20 billion parameters, reporting a zero-shot FID of 7.23 on MS-COCO.

Class-conditional ImageNet generation in the Taming Transformers paper reached an FID of 5.20 at 256 by 256 pixels when combined with rejection sampling, a strong score for an autoregressive model at the time.

VQGAN 和 Codebook 影像合成的未來

Research since 2022 has focused on the tokenizer as the bottleneck. ViT-VQGAN (2021) showed that better codebook learning raises downstream image quality, and Finite Scalar Quantization (September 2023) argued that the learned codebook can be replaced with simple rounding while avoiding codebook collapse and the extra losses VQ needs. Parallel decoding methods such as MaskGIT (2022) addressed the speed of sequential sampling, and Parti (2022) showed autoregressive models on VQ tokens can scale to 20 billion parameters. Open questions remain about how much compression is too much: higher downsampling makes sequences shorter and cheaper but loses fine detail, and there is no settled rule for choosing the factor per domain. A second open question is whether discrete tokens or continuous latents are the better interface for models that handle text and images together, since latent diffusion (2021) showed that continuous latents also cut compute sharply, while discrete tokens slot directly into the same sequence as text. Evaluation also lags, because FID and Inception Score capture distribution statistics rather than prompt faithfulness or the legibility of small details.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXiv (Esser, Rombach, Ommer)Taming Transformers for High-Resolution Image Synthesis
  2. arXiv (van den Oord, Vinyals, Kavukcuoglu)Neural Discrete Representation Learning
  3. arXiv (Google Research)Vector-quantized Image Modeling with Improved VQGAN
  4. arXiv / ECCV 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
  5. arXiv (Google Research)MaskGIT: Masked Generative Image Transformer
  6. arXiv (Google Research)Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
  7. arXiv / CVPR 2022使用潛在擴散模型的高解析度影像合成
  8. arXiv (Google DeepMind)Finite Scalar Quantization: VQ-VAE Made Simple

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the VQGAN and Codebook Image Synthesis quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

What is VQGAN and Codebook Image Synthesis?

VQGAN is a neural network that compresses an image into a small grid of entries from a learned codebook, a dictionary of reusable visual parts, and decodes that grid back into a sharp picture. Introduced in 2020, it made transformer image generation practical at high resolution and its tokenizer idea still underpins many text-to-image and multimodal models.

VQGAN 使用什麼將影像表示為離散標記?

VQGAN 將編碼器特徵量化為學習碼本中最近的條目,將影像轉換為離散碼索引序列。

What did the original VQGAN paper report gaining from adversarial and perceptual training?

The original paper reports that adversarial and perceptual losses helped maintain perceptual quality at increased compression in its evaluated settings. This is not a universal claim about every VQVAE or every token-grid size.

一旦影像是標記序列,什麼模型會產生新影像?

由於影像成為離散的標記序列,因此轉換器可以對它們進行自回歸建模,就像生成文字一樣。

什麼技術讓梯度流經不可微的量化步驟?

向量量化是不可微的,因此 VQGAN 使用直通梯度估計器來訓練編碼器。

VQGAN在2021年助力打造了哪些著名的AI藝術潮流?

將 VQGAN 與 CLIP 指導配對產生了廣泛共享的“VQGAN + CLIP”藝術,從而普及了文字驅動的圖像生成。