Stable Diffusion: How the Open Text-to-Image Model Works

Stable Diffusion is a text-to-image model: you type a description, and it produces an image that matches it. It belongs to a class called diffusion models, which learn to turn random noise into a picture one small step at a time. What made it different from earlier systems such as DALL-E 2 was not only quality but access. Stability AI released the first version publicly on August 22, 2022 under the CreativeML OpenRAIL-M license, which allows commercial and non-commercial use with use-based restrictions, and the weights could be downloaded and run locally. Stability's announcement put the memory need at about 6.9 GB of graphics card memory (VRAM).

The research behind it is the latent diffusion model, described by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Bjorn Ommer in 'High-Resolution Image Synthesis with Latent Diffusion Models', first posted to arXiv on December 20, 2021 and published at CVPR 2022. The paper's key idea is to run the expensive diffusion process inside the latent space of a pretrained autoencoder rather than on full-resolution pixels. An autoencoder is a pair of networks: an encoder that compresses an image into a smaller grid of numbers, called a latent, and a decoder that reconstructs the image from it. The authors argued this keeps visual detail while cutting compute, noting that pixel-space diffusion models often need hundreds of GPU days to train.

Generation works in a few distinct stages. First, a text encoder turns the prompt into a sequence of vectors. In version 1, this is a frozen CLIP ViT-L/14 text encoder with 123 million parameters, according to the CompVis model card. Second, the system starts from a latent filled with random noise. Third, a U-Net, a convolutional network with 860 million parameters in version 1, predicts the noise present in that latent, and a sampler removes a portion of it. The U-Net reads the text vectors through cross-attention layers, which let each region of the image look up the words most relevant to it. This loop repeats for a set number of steps, 50 in the reference script. Finally, the autoencoder's decoder converts the clean latent back into pixels. Because the autoencoder downsamples by a factor of 8, a 512 by 512 image is generated as a 64 by 64 latent.

Prompt adherence is controlled by classifier-free guidance, introduced by Jonathan Ho and Tim Salimans in July 2022. During training, the text condition is sometimes dropped, so the same network learns both a conditional and an unconditional prediction. At sampling time the two predictions are blended, pushing the result further in the direction the text suggests. The CompVis model card notes that checkpoints v1-3 and v1-4 were trained with 10% text-conditioning dropout for this reason, and the reference script uses a guidance scale of 7.5. Higher scales follow the prompt more literally at the cost of variety.

The training data came from LAION, a nonprofit that publishes large image-text datasets scraped from the web. Per the model card, v1-1 trained for 237,000 steps at 256 by 256 on laion2B-en and then 194,000 steps at 512 by 512 on a high-resolution subset, while later checkpoints continued on 'laion-aesthetics v2 5+', a subset filtered for images a classifier rated as more aesthetic. Stability AI provided compute, Runway collaborated, and LAION supported the work.

Later versions changed the architecture substantially, so claims about 'Stable Diffusion' should name a version. SDXL, described in a paper posted July 4, 2023, uses a U-Net about three times larger than earlier versions, a second text encoder, training on multiple aspect ratios, and an optional refinement model. Stable Diffusion 3, described in 'Scaling Rectified Flow Transformers for High-Resolution Image Synthesis' (March 5, 2024), replaced the U-Net with a transformer that keeps separate weights for image and text tokens and lets information flow in both directions. It also moved from standard diffusion to rectified flow, a formulation that connects data and noise along a straight line. The authors reported improved typography, text comprehension and human preference ratings.

Compared with closed services such as DALL-E or Midjourney, the main difference is control. With local weights, users can fine-tune on their own images, add control signals such as edge maps, run without per-image fees, and inspect the model. The trade-offs are that the user must supply hardware, manage safety settings, and accept that quality depends heavily on version, sampler and settings. Stability's 2022 release shipped a safety classifier enabled by default, but in a local install it can be adjusted or removed, which raised concern about misuse.

The limitations matter in practice. Version 1 was pretrained at 256 by 256 and finetuned at 512 by 512, so that is the size it was built for, and SDXL's move to multiple aspect ratios and SD3's focus on typography and text comprehension show which weaknesses the developers themselves set out to fix. Because the training images were scraped from the web, the model inherits whatever is over- or under-represented in that data. Copyright remains contested: in Getty Images v Stability AI, decided by the UK High Court in November 2025, Getty dropped its training claim for lack of a UK connection, the judge held the model was not an infringing copy because its weights do not store copies of Getty's images, and a small number of generated Getty watermarks were found to infringe trademarks. Other jurisdictions and cases may reach different conclusions.