VolgendeVolgende gids
Convolutionele neurale netwerken
Grondbeginselen
Basisprincipes GIDS
Scaling laws are empirical relationships between language-model loss and quantities such as parameters, training data, and compute.
Luister naar deze gidsAI-verteld · ~2 min
They can help researchers plan experiments and extrapolate within studied regimes, but they do not guarantee an exact result for a new architecture, dataset, or training setup.
Kaplan and colleagues’ 2020 study found empirical power-law relationships between language-model cross-entropy loss and model size, dataset size, and training compute over the regimes they tested. On log-log plots, a power law appears approximately linear. These fitted relationships can be used to estimate outcomes for larger runs, but extrapolations are conditional on comparable training and data setups. The original paper found large models to be more sample-efficient in its experiments and suggested compute-optimal training with relatively large models and fewer tokens than later Chinchilla work recommended. Later studies, including Llama 3, explored training beyond that earlier compute-optimal allocation.
Fitting a loss curve from smaller controlled runs to inform a larger training-budget decision
Comparing parameter and data allocations under a fixed pretraining-compute budget
Checking whether a forecast remains accurate when the data mixture or architecture changes
Separating measured loss trends from downstream capability claims in a model report
Scaling-law research studies new data mixtures, modalities, architectures, and inference-time computation. As training and data regimes change, researchers must remeasure the relevant curves. Scaling laws support resource planning, while downstream accuracy, safety, and reliability require separate evaluation.
Gratis nieuwsbrief
Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.
Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.
Test jezelf
Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.
Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie
Scaling laws are empirical relationships between language-model loss and quantities such as parameters, training data, and compute. They can help researchers plan experiments and extrapolate within studied regimes, but they do not guarantee an exact result for a new architecture, dataset, or training setup.
Kaplan et al. report empirical power-law relationships over their tested ranges; the fit should not be read as a guarantee outside comparable settings.
Kaplan et al. bestudeerde verlies als een machtswet in het aantal parameters (N), trainingsfiches (D) en totale berekeningen (C).
Fitted curves can inform large-run forecasts from smaller experiments, but changes in data, architecture, or training conditions can limit extrapolation.
Verlies heeft een reduceerbaar deel dat kleiner wordt naarmate de schaal groter wordt, plus een onherleidbaar minimum dat wordt bepaald door de inherente willekeur van de gegevens.
With L(X)=E+A·X^-α, a fixed multiplicative increase gives a similar fractional change in the reducible term, but smaller absolute changes as that term shrinks.
Blijf leren
Er zijn meer handleidingen voor dit onderwerp geselecteerd
VolgendeVolgende gids
Convolutionele neurale netwerken
Grondbeginselen