점수 기반 생성 모델

A generative model learns to produce new examples that look like its training data, such as faces or photographs. Most approaches try to learn the probability density of the data, a function that says how likely any given image is. That is hard because a valid density must integrate to one, which requires a normalizing constant that is intractable for complex neural networks. Score-based generative models sidestep the problem. Instead of the density itself, they learn its score: the gradient of the log density with respect to the data, written as the gradient of log p(x). At any point in image space, the score is an arrow pointing toward regions where realistic images are more likely, and the normalizing constant disappears when the gradient is taken.

The method has two halves. Training uses score matching, a technique introduced by Aapo Hyvarinen in the Journal of Machine Learning Research in 2005, which fits a model's log-density gradient to the data without ever computing the normalizing constant. In practice, the common variant is denoising score matching: add Gaussian noise to a training image, and train a neural network to predict the direction back toward the clean image, which equals the score of the noisy distribution. Sampling then uses Langevin dynamics, a procedure from physics. Start from random noise, take a small step in the direction of the predicted score, add a little fresh noise, and repeat many times. Under the right conditions the samples converge to the data distribution.

The naive version fails, and the fix defines the field. Real images occupy a tiny region of the space of all possible pixel arrays, so most of the space has almost no data. A score model trained only on clean data learns accurate arrows near real images but gets almost no training signal elsewhere, and Langevin sampling starting from random noise begins exactly in those poorly learned regions. Song and Ermon's 2019 NeurIPS paper, "Generative Modeling by Estimating Gradients of the Data Distribution", solved this by perturbing data with many noise levels at once. Large noise spreads data over the whole space so the arrows are informative everywhere; small noise keeps them precise near real images. A single Noise Conditional Score Network learns the score for every level, and annealed Langevin dynamics samples from the noisiest level down to the cleanest. On CIFAR-10 they used 10 noise levels in a geometric sequence from 1 down to 0.01, with 100 Langevin steps per level, and reported an Inception Score of 8.87 and an FID of 25.32.

The key milestone came in late 2020. In "Score-Based Generative Modeling through Stochastic Differential Equations", posted in November 2020 and given an Outstanding Paper Award at ICLR 2021, Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Ermon and Ben Poole replaced the finite list of noise levels with a continuous process. A stochastic differential equation (SDE), an equation describing a random process evolving over time, gradually turns data into noise. A known mathematical result says this process can be run backward in time if the score at every time is known, so learning the score is enough to turn noise back into data. The paper showed that both Song and Ermon's method and the Denoising Diffusion Probabilistic Models of Ho, Jain and Abbeel (2020) are special discretizations of this framework.

The unified view brought new tools. A predictor-corrector sampler alternates a numerical SDE solver step (the predictor) with a Langevin step (the corrector) to fix discretization errors. A deterministic "probability flow" ordinary differential equation shares the same intermediate distributions as the SDE, which allows exact likelihood computation and faster solvers. The paper reported an Inception Score of 9.89, an FID of 2.20 and a likelihood of 2.99 bits per dimension on CIFAR-10, and generated 1,024 by 1,024 face images.

Compared with alternatives, score-based models trade speed for stability and quality. Generative adversarial networks (GANs) produce an image in one forward pass but train a generator against a discriminator, a game that can be unstable and can drop parts of the data distribution. Variational autoencoders and normalizing flows give likelihoods but historically produced blurrier or less detailed images, or required restricted architectures. Score-based training is a simple regression problem with no adversary, and the 2019 paper noted it needs no special architecture constraints. The cost is sampling: generating one image requires hundreds to thousands of network evaluations. The SDE paper's samplers commonly used 1,000 or 2,000 steps.

The main limitations follow from that. Slow sampling makes deployment expensive, which is why much later research targets fewer steps. Score estimates are least accurate where data is sparse, so errors accumulate along the sampling path, and too few steps produce artifacts. Results depend on the noise schedule, the weighting of noise levels in the loss and the sampler, choices that interact in ways that are still tuned empirically. Like other generative models trained on large datasets, they can memorize and reproduce training examples, and the 2019 paper itself checked nearest neighbors in the training set to show it was not simply copying images.

In practice, score networks are usually U-Net-style convolutional networks conditioned on the noise level or time. The 2019 NCSN used a RefineNet variant and trained for 200,000 iterations with batch size 128 using the Adam optimizer. Training cost resembles that of other image models of the same size; the extra expense is at inference, where the network must run once per sampling step.