Scaling Laws
Empirical power-law relationships that predict how model performance improves as compute, data, and parameters increase, showing logarithmic capability gains for exponential resource investments.
Scaling laws are mathematical relationships discovered through empirical research that describe how language model performance (measured by loss) improves predictably as training compute, dataset size, and parameter count increase. The key insight is that these relationships follow power laws: capability gains are logarithmic while resource costs are exponential, meaning each doubling of performance requires more than double the investment. Major papers include OpenAI's Kaplan scaling laws (2020) and DeepMind's Chinchilla paper (2022), which disagreed on optimal resource allocation but confirmed the fundamental relationships.
Also known as
neural scaling laws, compute scaling, Kaplan scaling