Muse Masked Generative Imaging

Muse separates image representation, text conditioning, and token prediction. Its VQGAN tokenizers encode images into grids of discrete codebook tokens and decode token grids back into pixels. The text prompt is represented by a pretrained, frozen T5-XXL encoder. Frozen means its weights are not updated in this image-model training; it does not mean the encoder has no computational or memory cost. During training, the image model predicts masked tokens using the text representation and available unmasked image tokens. Generation begins with masked positions and proceeds through multiple decoding rounds. Several positions can be predicted during one round, so this is different from an autoregressive generator that emits one new image token after another. The paper's schedule commits a fraction of high-confidence predictions as unmasked while leaving other positions for subsequent rounds. This process still uses repeated inference steps: the full image is not guaranteed to resolve in a single pass. A base model handles a lower-resolution token grid, and a super-resolution model predicts a higher-resolution representation. The tokenizer decoder then produces the image pixels. This architecture also supports image editing, including inpainting, outpainting, and demonstrations of prompt-based mask-free editing. The reported comparisons explain the authors' efficiency findings for evaluated configurations. They should not be turned into a claim that Muse is faster than every diffusion implementation or that a frozen text encoder is free. Model size, decoding settings, resolution, and hardware all matter when assessing an actual generation workload.