AI Glossary Term
What does “Scaling Law” mean?
An empirical relationship showing how performance improves with model size, data, or compute.
How Scaling Law works
Example
Researchers trained many models of different sizes and found that test loss decreased in a predictable pattern as size, data, and compute increased. That let them estimate the results of a bigger run before paying for it.
Why it matters
Scaling laws guide how labs spend training budgets. They are empirical trends, not guarantees, and they may not hold forever.
Common questions
Who found them?
Studies from OpenAI in 2020 and DeepMind's 2022 'Chinchilla' work are among the best-known.
Does bigger always win?
Within a regime, loss tends to improve, but returns diminish and data quality matters.
What does Chinchilla say?
That, for a given compute budget, many large models were too big for the amount of data used, and balancing the two works better.