Mifano zinazotegemea Nishati

An energy-based model, or EBM, is a way of describing what a model knows using a single scalar score. For any setting of the variables the model cares about, for example an image and a label, or a sentence and its translation, the model outputs a number called the energy. Low energy means the combination fits well together; high energy means it does not. The word comes from physics, where systems tend to settle into low-energy states. Yann LeCun and colleagues' 2006 "A Tutorial on Energy-Based Learning" put it simply: inference means fixing the observed variables and searching for values of the remaining variables that make the energy as small as possible, and learning means shaping the energy function so that correct configurations get lower energy than incorrect ones.

A concrete example makes this clearer. Suppose X is the pixels of a photo and Y is one of five labels: human, animal, airplane, car or truck. An EBM computes E(Y, X) for each of the five labels and predicts the label with the lowest energy. A standard neural network classifier already does something very close to this: its output scores, called logits, can be read as negative energies. What makes the energy view useful is that it does not require those scores to be turned into probabilities. A probabilistic model must divide by a normalizing constant, the sum or integral of the unnormalized scores over every possible configuration, so that probabilities add up to one. For images or sentences, that sum covers an astronomically large space and usually cannot be computed. The LeCun tutorial argues that because EBMs have no requirement for proper normalization, this problem is avoided and model designers gain flexibility in architectures and training criteria.

When a probability is needed, an energy can be converted into one using the Gibbs or Boltzmann distribution: the probability of a configuration is proportional to the exponential of minus its energy, divided by the normalizing constant, often written Z. Lower energy therefore means exponentially higher probability. This link is exact, so every probabilistic model with a positive density can be written as an EBM, and every EBM defines a probability distribution whenever Z is finite. The catch is that Z is still there; EBMs move the difficulty from defining the model to training and sampling it.

Training has to do two things at once: push energy down on real data and push it up on everything else. If a method only pushes energy down on the training examples, the model can cheat by giving every input low energy, a failure often called collapse. Song and Kingma's 2021 tutorial "How to Train Your Energy-Based Models" groups the main solutions into three families. Maximum likelihood with Markov chain Monte Carlo (MCMC) sampling draws "negative" samples from the current model and raises their energy. Score matching avoids sampling by fitting the gradient of the log density with respect to the data, which does not depend on Z. Noise contrastive estimation trains the model to tell real data apart from samples of a known noise distribution. The tutorial shows that these three approaches are theoretically connected.

The history runs through some of the best-known work in the field. The Nobel Committee's popular explanation of the 2024 Physics prize describes how John Hopfield's 1982 associative memory network was defined by an energy borrowed from models of magnetic spins: stored patterns are low-energy states, and the network recovers a pattern by updating its units so the energy falls. Geoffrey Hinton built on this with the Boltzmann machine, published in 1985, which uses tools from statistical physics to learn a probability distribution over patterns and can generate new examples. The two shared the 2024 Nobel Prize in Physics for these foundations. In 2006 LeCun's group published its tutorial, recasting conditional random fields, max-margin Markov networks and graph transformer networks as EBMs. In 2019 Du and Mordatch showed that deep convolutional EBMs trained with MCMC could generate CIFAR-10 and ImageNet images approaching the quality of generative adversarial networks of that time, and Grathwohl and colleagues showed that a standard classifier could be reinterpreted and trained as an EBM. The same year Song and Ermon's score-based model, which learns energy gradients rather than energies, helped set up the diffusion models that now dominate image generation.

Compared with other generative approaches, EBMs make the fewest structural assumptions. Autoregressive models such as GPT-style language models factor the probability into a product of next-token probabilities, which makes exact likelihood easy but fixes an order. Normalizing flows require invertible architectures so the density can be computed exactly. Variational autoencoders train an approximate lower bound. Generative adversarial networks generate samples quickly but do not provide a density at all. An EBM can use any neural network that outputs one number, can combine several energies by adding them, which Du and Mordatch used for compositional generation, and gives a score that can flag unusual inputs. Diffusion and score-based models keep much of this flexibility while sidestepping the hardest part of EBM training by learning gradients at many noise levels.

The weaknesses are real. Sampling from an EBM usually requires many iterative MCMC steps, often Langevin dynamics, which repeatedly moves a sample a small step down the energy gradient and adds noise. That makes both training and generation slow, and short chains can leave the model poorly trained in regions the sampler never visits. Training can be unstable, and because Z is unknown, the exact likelihood of a test example usually cannot be reported, which makes comparison with other models harder. Results are sensitive to step sizes, noise levels and regularization.

In practice, building a small EBM requires no special hardware beyond what a classifier of the same size would need, but training costs multiply with the number of sampling steps per update. For a learner, the most accessible routes are reading a classifier's logits as energies, which costs nothing extra, or training a score-based model, which turns the energy idea into a stable recipe with standard deep learning tools.