概述
It aims to reduce the objective locally, while step size and loss shape determine whether a particular update succeeds.
深入探讨
For a differentiable scalar objective, the gradient identifies the direction of steepest local increase under the usual Euclidean norm. Plain gradient descent moves in the opposite direction. Its update is new weight equals old weight minus learning rate times gradient. In a scalar illustration with old weight two, gradient three, and learning rate one-tenth, the new weight is one-point-seven. That formula describes the proposed update, not a guarantee that every finite step reduces loss. A learning rate that is too large can overshoot or lead to divergence, and a very small rate can make progress slow. Full-batch gradient descent uses the selected training objective over the whole dataset. Stochastic and mini-batch methods instead use gradients from sampled examples or batches. The batch gradient is exact for that batch calculation but estimates the full-data gradient; the resulting variability can help or hurt depending on the problem. Momentum combines information from earlier gradients to influence the current update. Adam maintains per-parameter moment estimates that adapt updates, so it should not be confused with plain unmodified gradient descent. Nonconvex loss surfaces also contain stationary points. A zero gradient is not proof of a globally lowest objective: local minima, maxima, and saddle points can have zero gradient. Training therefore needs monitoring of the objective and relevant evaluation metrics rather than assuming that a named optimizer or a stationary point establishes success.
现实世界的实施
使用小批量更新降低语言模型在数十亿个训练令牌中的预测误差
调整学习率，使图像模型快速收敛，而不会导致损失爆炸
利用动量加速陷入狭长损失谷的语音识别网络的训练
应用 Adam 在小数据集上微调模型，其中每个参数的学习率有助于稳定性
梯度下降的未来
Optimization research studies schedules, moment estimates, curvature, and distributed implementations under specific objectives and workloads. A method that performs well in one experiment may require different settings elsewhere. Report learning rates, sampling, state rules, and evaluation conditions when comparing training runs. Verify that updates change parameters as intended and track whether useful metrics improve; a direction motivated by local calculus is not a universal guarantee of convergence or application quality.
付诸实践
为什么这很重要
- 更清晰的判决. 它可以帮助您将清晰的技术声明与营销语言分开。
- 成本与预算. 在花费金钱或时间之前，您可以提出更好的实施问题。
- 团队与工作流程. 具有共同理解的团队可以做出更好的产品、政策和学习决策。
留意
- 不同的团队可能会以不同的方式使用同一术语，因此请尽早定义范围。
- 基准测试可能看起来很强大，但实际性能却参差不齐。
- 忽视数据质量和评估计划通常会产生脆弱的结果。
后续步骤
- 从您需要的结果的简单语言定义开始。
- 在测试之前选择一种成功指标和一种失败条件。
- 使用代表性数据运行小型试点，而不是完善的演示集。
- 记录梯度下降在哪些方面有帮助以及在哪些方面更简单的方法更好。
不断探索
常见问题
什么是梯度下降？
Plain gradient descent updates parameters by subtracting a learning rate times the objective's gradient. It aims to reduce the objective locally, while step size and loss shape determine whether a particular update succeeds.
在普通梯度下降中，相对于目标梯度使用哪个方向？
Plain gradient descent subtracts a scaled gradient, with improvement dependent on step size and objective shape.
学习率控制什么？
学习率衡量每次更新时权重移动的距离；太大的超调，太小的训练会变得非常缓慢。
小批量梯度与完整选择的训练数据目标有何关系？
The computed batch derivative is exact for that batch, while sampling makes it an estimate of the full-data derivative.
学习率太大会导致什么问题？
大的步骤可能会跳过最小值并反弹或爆炸，导致损失增加而不是稳定。
动量会在以后的梯度下降更新中携带哪些信息？
Momentum combines gradient history, which can affect progress and oscillation; its effect still depends on settings.