рдмреБрдирд┐рдпрд╛рджреА рдЧрд╛рдЗрдб

Empirical Risk Minimization

Empirical risk minimization (ERM) chooses a hypothesis by minimizing its average loss on a finite training sample as a proxy for expected loss under an unknown data distribution.

рдЗрд╕ рдЧрд╛рдЗрдб рдХреЛ рд╕реБрдиреЗрдВрдПрдЖрдИ-рдиреИрд░реЗрдЯреЗрдб ┬╖ ~3 min

рдЗрд╕ рдкреГрд╖реНрда рдкрд░3 рдорд┐рдирдЯ рд▓рд╛рд▓
  1. рд╕рд┐рдВрд╣рд╛рд╡рд▓реЛрдХрди
  2. рдЧрд╣рд░рд╛ рдЧреЛрддрд╛
  3. рд╡рд╛рд╕реНрддрд╡рд┐рдХ рд╡рд┐рд╢реНрд╡ рдХрд╛рд░реНрдпрд╛рдиреНрд╡рдпрди
  4. The Future of Empirical Risk Minimization
  5. рдЗрд╕реЗ рд╡реНрдпрд╡рд╣рд╛рд░ рдореЗрдВ рд▓рд╛рдирд╛
  6. рдЕрдиреНрд╡реЗрд╖рдг рдХрд░рддреЗ рд░рд╣реЗрдВ
  7. рдЕрдХреНрд╕рд░ рдкреВрдЫреЗ рдЬрд╛рдиреЗ рд╡рд╛рд▓реЗ рдкреНрд░рд╢реНрдиреЛрдВ

рд╕рд┐рдВрд╣рд╛рд╡рд▓реЛрдХрди

It matters because fit to that sample can differ from performance on new data, which is why model complexity and independent evaluation are part of the learning problem.

рдЧрд╣рд░рд╛ рдЧреЛрддрд╛

Empirical risk minimization formalizes what most machine learning training actually does. The true risk of a model is the expected value of some loss function over the entire, usually infinite and unknown, distribution of possible data. Since that distribution is inaccessible, ERM substitutes it with the empirical risk: the average loss computed over a finite training sample drawn from that distribution. Learning becomes an optimization problem: search over a hypothesis class (all linear functions, or all neural networks of a given architecture) for the one that minimizes this average sample loss, typically via gradient descent or a closed-form solution. For a fixed hypothesis and representative independent samples, average sample loss can estimate expected loss. To make a guarantee that holds across a whole hypothesis class, additional conditions on the class and sample are needed; uniform-convergence results such as VC bounds make those assumptions explicit. The catch is that minimizing training loss too aggressively, especially with a hypothesis class that is complex relative to the sample size, can drive empirical risk toward zero while true risk stays high; the model has memorized noise specific to the training sample rather than learning the underlying pattern. This is overfitting, and it is why practitioners use regularization, cross-validation, and held-out test sets: not because ERM is wrong, but because minimizing empirical risk alone provides no guarantee about true risk without additional constraints or enough data relative to model complexity. A common misconception is that ERM refers to a specific algorithm; it is a general principle that many learning procedures use, sometimes with regularization or other constraints and different loss functions or hypothesis classes.

рд╡рд╛рд╕реНрддрд╡рд┐рдХ рд╡рд┐рд╢реНрд╡ рдХрд╛рд░реНрдпрд╛рдиреНрд╡рдпрди

Training a spam filter by adjusting its parameters to minimize the fraction of misclassified emails in a labeled training set, hoping this generalizes to new incoming email.

Fitting a linear regression line by minimizing mean squared error across the observed data points, which is literally ERM with squared-error loss.

A neural network's training loop that repeatedly computes gradient updates to reduce average cross-entropy loss over mini-batches drawn from the training set.

A model that achieves near-zero error on training data but performs poorly on new data, illustrating the gap between empirical risk (training loss) and true risk (expected loss on unseen data), i.e. overfitting.

The Future of Empirical Risk Minimization

ERM remains a widely used training principle across supervised learning, often combined with regularization or other constraints, from simple regressions to large neural networks, and this is unlikely to change since it is a general mathematical framework rather than a specific technique that could be superseded. Active research continues on refining the gap between empirical and true risk for very large, over-parameterized models, where classical overfitting intuitions sometimes fail to predict observed generalization behavior. Work on distributionally robust optimization and other risk formulations extends ERM's ideas to handle shifts between training and deployment data, but these remain extensions of the same underlying principle.

рдЗрд╕реЗ рд╡реНрдпрд╡рд╣рд╛рд░ рдореЗрдВ рд▓рд╛рдирд╛

рдпрд╣ рдХреНрдпреЛрдВ рдорд╛рдпрдиреЗ рд░рдЦрддрд╛ рд╣реИ?

  • рд╕реНрдкрд╖реНрдЯ рдирд┐рд░реНрдгрдп. рдпрд╣ рдЖрдкрдХреЛ рд╕реНрдкрд╖реНрдЯ рддрдХрдиреАрдХреА рджрд╛рд╡реЛрдВ рдХреЛ рдорд╛рд░реНрдХреЗрдЯрд┐рдВрдЧ рднрд╛рд╖рд╛ рд╕реЗ рдЕрд▓рдЧ рдХрд░рдиреЗ рдореЗрдВ рдорджрдж рдХрд░рддрд╛ рд╣реИред
  • рд▓рд╛рдЧрдд рдФрд░ рдмрдЬрдЯ. рдЖрдк рдкреИрд╕рд╛ рдпрд╛ рд╕рдордп рдЦрд░реНрдЪ рдХрд░рдиреЗ рд╕реЗ рдкрд╣рд▓реЗ рдмреЗрд╣рддрд░ рдХрд╛рд░реНрдпрд╛рдиреНрд╡рдпрди рд╕рдВрдмрдВрдзреА рдкреНрд░рд╢реНрди рдкреВрдЫ рд╕рдХрддреЗ рд╣реИрдВред
  • рдЯреАрдо рдФрд░ рд╡рд░реНрдХрдлрд╝реНрд▓реЛ. рд╕рд╛рдЭрд╛ рд╕рдордЭ рд╡рд╛рд▓реА рдЯреАрдореЗрдВ рдмреЗрд╣рддрд░ рдЙрддреНрдкрд╛рдж, рдиреАрддрд┐ рдФрд░ рд╕реАрдЦрдиреЗ рдХреЗ рдирд┐рд░реНрдгрдп рд▓реЗрддреА рд╣реИрдВред

рдХрд╛ рдзреНрдпрд╛рди рд░рдЦреЗрдВ

  • рдЕрд▓рдЧ-рдЕрд▓рдЧ рдЯреАрдореЗрдВ рдПрдХ рд╣реА рд╢рдмреНрдж рдХрд╛ рдЕрд▓рдЧ-рдЕрд▓рдЧ рдЗрд╕реНрддреЗрдорд╛рд▓ рдХрд░ рд╕рдХрддреА рд╣реИрдВ, рдЗрд╕рд▓рд┐рдП рджрд╛рдпрд░реЗ рдХреЛ рдкрд╣рд▓реЗ рд╣реА рдкрд░рд┐рднрд╛рд╖рд┐рдд рдХрд░ рд▓реЗрдВред
  • рдмреЗрдВрдЪрдорд╛рд░реНрдХ рдордЬрдмреВрдд рджрд┐рдЦ рд╕рдХрддреЗ рд╣реИрдВ рдЬрдмрдХрд┐ рд╡рд╛рд╕реНрддрд╡рд┐рдХ рджреБрдирд┐рдпрд╛ рдХрд╛ рдкреНрд░рджрд░реНрд╢рди рдЕрд╕рдорд╛рди рд╣реИред
  • рдбреЗрдЯрд╛ рдЧреБрдгрд╡рддреНрддрд╛ рдФрд░ рдореВрд▓реНрдпрд╛рдВрдХрди рдпреЛрдЬрдирд╛рдУрдВ рдХреА рдЕрдирджреЗрдЦреА рдЕрдХреНрд╕рд░ рдирд╛рдЬреБрдХ рдкрд░рд┐рдгрд╛рдо рдкреИрджрд╛ рдХрд░рддреА рд╣реИред

рдЕрдЧрд▓реЗ рдЪрд░рдг

  1. рдЖрдкрдХреЛ рдЬрд┐рд╕ рдкрд░рд┐рдгрд╛рдо рдХреА рдЖрд╡рд╢реНрдпрдХрддрд╛ рд╣реИ рдЙрд╕рдХреА рд╕рд░рд▓ рднрд╛рд╖рд╛ рдореЗрдВ рдкрд░рд┐рднрд╛рд╖рд╛ рд╕реЗ рд╢реБрд░реБрдЖрдд рдХрд░реЗрдВред
  2. рдкрд░реАрдХреНрд╖рдг рд╕реЗ рдкрд╣рд▓реЗ рдПрдХ рд╕рдлрд▓рддрд╛ рдореАрдЯреНрд░рд┐рдХ рдФрд░ рдПрдХ рд╡рд┐рдлрд▓рддрд╛ рд╕реНрдерд┐рддрд┐ рдЪреБрдиреЗрдВред
  3. рдкреНрд░рддрд┐рдирд┐рдзрд┐ рдбреЗрдЯрд╛ рдХреЗ рд╕рд╛рде рдПрдХ рдЫреЛрдЯрд╛ рдкрд╛рдпрд▓рдЯ рдЪрд▓рд╛рдПрдБ, рди рдХрд┐ рдПрдХ рдкрд░рд┐рд╖реНрдХреГрдд рдбреЗрдореЛ рд╕реЗрдЯред
  4. Document where Empirical Risk Minimization helps and where simpler methods are better.

рдЕрдиреНрд╡реЗрд╖рдг рдХрд░рддреЗ рд░рд╣реЗрдВ

рдореБрдлрд╝реНрдд рдиреНрдпреВрдЬрд╝рд▓реЗрдЯрд░

рджреИрдирд┐рдХ рдПрдЖрдИ рдмреНрд░реАрдлрд┐рдВрдЧ рдкреНрд░рд╛рдкреНрдд рдХрд░реЗрдВ

рд╣рд░ рдХрд╛рд░реНрдпрджрд┐рд╡рд╕ рдХреА рд╕реБрдмрд╣ рддреАрди рд╕рддреНрдпрд╛рдкрд┐рдд рдПрдЖрдИ рдХрд╣рд╛рдирд┐рдпрд╛рдБ, рд╕рд╛рджреЗ рдЕрдВрдЧреНрд░реЗрдЬреА рдореЗрдВ рд▓рд┐рдЦреА рдЧрдИрдВред рд╣рдореЗрд╢рд╛ рдХреЗ рд▓рд┐рдП рдореБрдлрд╝реНрдд, рдХреЛрдИ рд╡рд┐рдЬреНрдЮрд╛рдкрди рдирд╣реАрдВред

рдкреНрд░рддреНрдпреЗрдХ рд╕рдкреНрддрд╛рд╣ рдХреЗ рджрд┐рди рдПрдХ рдИрдореЗрд▓. рдПрдХ рдХреНрд▓рд┐рдХ рдореЗрдВ рд╕рджрд╕реНрдпрддрд╛ рд╕рдорд╛рдкреНрдд рдХрд░реЗрдВ. рд╣рдо рдЖрдкрдХрд╛ рдкрддрд╛ рдХрднреА рдирд╣реАрдВ рдмреЗрдЪрддреЗ рдпрд╛ рд╕рд╛рдЭрд╛ рдирд╣реАрдВ рдХрд░рддреЗред

рд╕реНрд╡рдпрдВ рдХреА рдЬрд╛рдВрдЪ рдХрд░реЛ

Take the Empirical Risk Minimization quiz

рдкреНрд░рддреНрдпреЗрдХ рдЙрддреНрддрд░ рдкрд░ рддреНрд╡рд░рд┐рдд рдкреНрд░рддрд┐рдХреНрд░рд┐рдпрд╛, рдФрд░ рдкрд╛рдареНрдпрдХреНрд░рдо рдкрд╛рд╕ рдХрд░рдиреЗ рдХреЗ рдмрд╛рдж рдПрдХ рд╕рддреНрдпрд╛рдкрди рдпреЛрдЧреНрдп рдЖрдИрдбреА рдХреЗ рд╕рд╛рде рдПрдХ рд╕рд╛рдЭрд╛ рдХрд░рдиреЗ рдпреЛрдЧреНрдп рдкреНрд░рдорд╛рдгрдкрддреНрд░ред

рдкреНрд░рд╢реНрдиреЛрддреНрддрд░реА рдкреНрд░рд╛рд░рдВрдн рдХрд░реЗрдВ

рдирд┐рдГрд╢реБрд▓реНрдХ рдПрдЖрдИ рд╢рд┐рдХреНрд╖рд╛ рдХрд╛ рд╕рдорд░реНрдерди рдХрд░реЗрдВред AI Understanding рдПрдХ 501(c)(3) рдЧреИрд░-рд▓рд╛рднрдХрд╛рд░реА рд╕рдВрд╕реНрдерд╛ рд╣реИ - рдХреЛрдИ рд╡рд┐рдЬреНрдЮрд╛рдкрди рдирд╣реАрдВ, рдХреЛрдИ рдкреЗрд╡реЙрд▓ рдирд╣реАрдВред рджрд╛рди рдХрд░реЗрдВ

рдЕрдХреНрд╕рд░ рдкреВрдЫреЗ рдЬрд╛рдиреЗ рд╡рд╛рд▓реЗ рдкреНрд░рд╢реНрдиреЛрдВ

What is Empirical Risk Minimization?

Empirical risk minimization (ERM) chooses a hypothesis by minimizing its average loss on a finite training sample as a proxy for expected loss under an unknown data distribution. It matters because fit to that sample can differ from performance on new data, which is why model complexity and independent evaluation are part of the learning problem.

What does empirical risk minimization actually minimize during training?

ERM substitutes the unobservable true risk with the empirical risk, the average loss measured on the finite training sample actually available.

Why can't a model directly minimize true risk instead of empirical risk?

True risk is an expectation over the full, typically unknown and infinite data distribution, so it cannot be computed directly; only a finite sample is available.

As an IID sample grows, what classical result helps explain why its average loss can estimate expected loss for a fixed model?

The guide cites the law of large numbers and VC-theory uniform convergence bounds as the theoretical basis for empirical risk converging to true risk with more data.

During training, what does overfitting look like when empirical and true risk are compared?

Overfitting occurs when a model memorizes training-sample-specific noise, achieving very low empirical risk while its true risk on unseen data stays high.

What role does regularization play in the ERM objective?

Regularization adds a penalty, such as an L2 norm term, to the ERM objective, accepting somewhat higher training loss in exchange for a hypothesis that generalizes better.

рд╕реАрдЦрддреЗ рд░рд╣реЛ

рд╕рдВрдмрдВрдзрд┐рдд рдорд╛рд░реНрдЧрджрд░реНрд╢рд┐рдХрд╛рдПрдБ

рдЗрд╕ рд╡рд┐рд╖рдп рдХреЗ рд▓рд┐рдП рдЕрдзрд┐рдХ рдорд╛рд░реНрдЧрджрд░реНрд╢рд┐рдХрд╛рдПрдБ рдЪреБрдиреА рдЧрдИрдВ

рдЖрдЧреЗрдЕрдЧрд▓реА рдЧрд╛рдЗрдб

рдиреНрдпреВрдирддрдо рдмреЗрдпрд╕ рдЬреЛрдЦрд┐рдо рдбрд┐рдХреЛрдбрд┐рдВрдЧ

рднрд╛рд╖рд╛ рдП.рдЖрдИ

рдирд┐рдГрд╢реБрд▓реНрдХ AI рдкреНрд░рдорд╛рдгрдкрддреНрд░ рдкреНрд░рд╛рдкреНрдд рдХрд░реЗрдВрдПрдХ рд╕рдВрдХреНрд╖рд┐рдкреНрдд рдирд┐рдГрд╢реБрд▓реНрдХ рдкрд╛рдареНрдпрдХреНрд░рдо рдФрд░ рдкрд░реАрдХреНрд╖рд╛ред рдХреЛрдИ рдХреАрдордд рдирд╣реАрдВ, рдХреЛрдИ рдХрд╛рд░реНрдб рдирд╣реАрдВ. рд▓рдЧрднрдЧ 2.5 рдШрдВрдЯреЗ.рдирд┐рдГрд╢реБрд▓реНрдХ рдкреНрд░рд╛рд░рдВрдн рдХрд░реЗрдВ тЖТ