AI Glossary Term
What does “Gradient Descent” mean?
An optimization method that updates parameters in the direction that reduces error.
How Gradient Descent works
Example
Picture standing on a foggy hillside and taking small steps in whichever direction slopes downward most steeply. Gradient descent does this with the model's loss, adjusting weights a little each step until the loss stops dropping.
Why it matters
Gradient descent and its variants are how nearly all neural networks are trained. Step size and data handling choices affect whether training succeeds.
Common questions
What is stochastic gradient descent?
A version that estimates the gradient from a small random batch of data each step, which is faster and works well in practice.
Can it get stuck?
It can slow near flat regions or settle in a local minimum, though in large networks these are usually less troublesome than expected.
What is Adam?
A popular variant that adapts step sizes per parameter using running averages of gradients.