概述
They can help researchers plan experiments and extrapolate within studied regimes, but they do not guarantee an exact result for a new architecture, dataset, or training setup.
深入探讨
Kaplan and colleagues’ 2020 study found empirical power-law relationships between language-model cross-entropy loss and model size, dataset size, and training compute over the regimes they tested. On log-log plots, a power law appears approximately linear. These fitted relationships can be used to estimate outcomes for larger runs, but extrapolations are conditional on comparable training and data setups. The original paper found large models to be more sample-efficient in its experiments and suggested compute-optimal training with relatively large models and fewer tokens than later Chinchilla work recommended. Later studies, including Llama 3, explored training beyond that earlier compute-optimal allocation.
现实世界的实施
Fitting a loss curve from smaller controlled runs to inform a larger training-budget decision
Comparing parameter and data allocations under a fixed pretraining-compute budget
Checking whether a forecast remains accurate when the data mixture or architecture changes
Separating measured loss trends from downstream capability claims in a model report
神经网络扩展定律的未来
Scaling-law research studies new data mixtures, modalities, architectures, and inference-time computation. As training and data regimes change, researchers must remeasure the relevant curves. Scaling laws support resource planning, while downstream accuracy, safety, and reliability require separate evaluation.
付诸实践
为什么这很重要
- 更清晰的判决. 它可以帮助您将清晰的技术声明与营销语言分开。
- 成本与预算. 在花费金钱或时间之前，您可以提出更好的实施问题。
- 团队与工作流程. 具有共同理解的团队可以做出更好的产品、政策和学习决策。
留意
- 不同的团队可能会以不同的方式使用同一术语，因此请尽早定义范围。
- 基准测试可能看起来很强大，但实际性能却参差不齐。
- 忽视数据质量和评估计划通常会产生脆弱的结果。
后续步骤
- 从您需要的结果的简单语言定义开始。
- 在测试之前选择一种成功指标和一种失败条件。
- 使用代表性数据运行小型试点，而不是完善的演示集。
- 记录神经网络缩放定律在哪些方面有帮助以及在哪些方面更简单的方法更好。
不断探索
常见问题
什么是神经网络的缩放定律？
Scaling laws are empirical relationships between language-model loss and quantities such as parameters, training data, and compute. They can help researchers plan experiments and extrapolate within studied regimes, but they do not guarantee an exact result for a new architecture, dataset, or training setup.
卡普兰等人采取了什么模式。随着模型大小、数据或计算量的增加，报告语言模型损失吗？
Kaplan et al. report empirical power-law relationships over their tested ranges; the fit should not be read as a guarantee outside comparable settings.
原始缩放定律与损失相关的三个量是什么？
卡普兰等人。研究了参数计数 (N)、训练标记 (D) 和总计算 (C) 中的幂律损失。
在规划更大规模的训练时，缩放法则拟合有何帮助？
Fitted curves can inform large-run forecasts from smaller experiments, but changes in data, architecture, or training conditions can limit extrapolation.
缩放定律损失公式中的常数（不可约）项代表什么？
损失有一个可减少的部分，随着规模的扩大而缩小，加上由数据固有的随机性设定的不可减少的下限。
为什么在幂律拟合下，随着计算量不断增加，绝对损失减少量会变得更小？
With L(X)=E+A·X^-α, a fixed multiplicative increase gives a similar fractional change in the reducible term, but smaller absolute changes as that term shrinks.