基礎知識指南

神經網路的縮放定律

縮放定律是語言模型損失與參數、訓練資料和計算等數量之間的經驗關係。

聆聽此指南AI 語音敘述 · ~2 min

本頁閱讀時間約2分鐘
  1. 概述
  2. 深入探討
  3. 現實世界的實施
  4. 神經網路擴展定律的未來
  5. 付諸實踐
  6. 不斷探索
  7. 常見問題

概述

They can help researchers plan experiments and extrapolate within studied regimes, but they do not guarantee an exact result for a new architecture, dataset, or training setup.

深入探討

Kaplan and colleagues’ 2020 study found empirical power-law relationships between language-model cross-entropy loss and model size, dataset size, and training compute over the regimes they tested. On log-log plots, a power law appears approximately linear. These fitted relationships can be used to estimate outcomes for larger runs, but extrapolations are conditional on comparable training and data setups. The original paper found large models to be more sample-efficient in its experiments and suggested compute-optimal training with relatively large models and fewer tokens than later Chinchilla work recommended. Later studies, including Llama 3, explored training beyond that earlier compute-optimal allocation.

現實世界的實施

Fitting a loss curve from smaller controlled runs to inform a larger training-budget decision

Comparing parameter and data allocations under a fixed pretraining-compute budget

Checking whether a forecast remains accurate when the data mixture or architecture changes

Separating measured loss trends from downstream capability claims in a model report

神經網路擴展定律的未來

Scaling-law research studies new data mixtures, modalities, architectures, and inference-time computation. As training and data regimes change, researchers must remeasure the relevant curves. Scaling laws support resource planning, while downstream accuracy, safety, and reliability require separate evaluation.

付諸實踐

為什麼這很重要

  • 更明確的決策. 它可以幫助您將清晰的技術聲明與行銷語言分開。
  • 成本與預算. 在花費金錢或時間之前,您可以提出更好的實施問題。
  • 團隊與工作流程. 具有共同理解的團隊可以做出更好的產品、政策和學習決策。

留意

  • 不同的團隊可能會以不同的方式使用相同術語,因此請儘早定義範圍。
  • 基準測試可能看起來很強大,但實際效能卻參差不齊。
  • 忽視數據品質和評估計劃通常會產生脆弱的結果。

後續步驟

  1. 從您需要的結果的簡單語言定義開始。
  2. 在測試之前選擇一種成功指標和一種失敗條件。
  3. 使用代表性資料運行小型試點,而不是完善的演示集。
  4. 記錄神經網路縮放定律在哪些方面有幫助以及在哪些方面更簡單的方法更好。

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the Scaling Laws for Neural Networks quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是神經網路的縮放定律?

Scaling laws are empirical relationships between language-model loss and quantities such as parameters, training data, and compute. They can help researchers plan experiments and extrapolate within studied regimes, but they do not guarantee an exact result for a new architecture, dataset, or training setup.

卡普蘭等人採取了什麼模式。隨著模型大小、資料或計算量的增加,報告語言模型損失嗎?

Kaplan et al. report empirical power-law relationships over their tested ranges; the fit should not be read as a guarantee outside comparable settings.

原始縮放定律與損失相關的三個量是什麼?

卡普蘭等。研究了參數計數 (N)、訓練標記 (D) 和總計計算 (C) 中的冪律損失。

在規劃更大規模的訓練時,縮放法則擬合有何幫助?

Fitted curves can inform large-run forecasts from smaller experiments, but changes in data, architecture, or training conditions can limit extrapolation.

縮放定律損失公式中的常數(不可約)項代表什麼?

損失有一個可減少的部分,隨著規模的擴大而縮小,加上由資料固有的隨機性設定的不可減少的下限。

為什麼在冪律擬合下,隨著計算量不斷增加,絕對損失減少量會變得更小?

With L(X)=E+A·X^-α, a fixed multiplicative increase gives a similar fractional change in the reducible term, but smaller absolute changes as that term shrinks.