Memorization in Diffusion Models

A diffusion model learns to reverse a noising process: during training it sees images with noise added and learns to predict and remove that noise. If it generalizes well, it learns the statistics of images in general and produces new combinations. But neural networks can also fit individual examples, and when one image appears hundreds or thousands of times in the data, the cheapest way to reduce training loss on it is to store it. The key evidence comes from Carlini and colleagues' 2023 paper 'Extracting Training Data from Diffusion Models'. They generated many images for captions of the most duplicated training images and flagged outputs that were extremely close to a training image. They extracted over a hundred near-identical copies from Stable Diffusion, including photos of identifiable people, and found higher memorization rates in Google's Imagen. Separately, Somepalli and colleagues (2022-2023) showed that Stable Diffusion outputs frequently copied parts or layouts of training images, and linked copying to duplicated images and highly specific captions. Three misconceptions are worth correcting. First, memorization is not the model's normal behavior: extracted copies are rare relative to all generations. Second, rare does not mean harmless, because a determined user can target the images most likely to be memorized. Third, memorization is not only exact pixel copying; models can reproduce characters, logos, watermarks and distinctive compositions. This evidence has featured in legal debates, including lawsuits such as Getty Images v. Stability AI and Andersen v. Stability AI, although how courts treat model training and outputs is still being worked out and varies by country. For privacy, the concern is sharper in small fine-tuning datasets, such as medical or personal photos, where each example has more influence.