12 — Architecture

Scaling Laws#

Chinchilla · Power LawsStatistical 1 min read
◆ The PatternPredicting performance before spending millions on compute

Scaling laws reveal that LLM loss follows power laws in parameters (N), data (D), and compute (C). The Chinchilla paper showed that most models were trained on too little data — the optimal ratio is roughly 20 tokens per parameter.

L(N) ≈ (N_c / N)^α_N L(D) ≈ (D_c / D)^α_D
Loss decreases as a power law. α_N ≈ 0.076, α_D ≈ 0.095 from Kaplan et al.
Chinchilla optimal: D_opt ≈ 20 · N
For a 70B model, train on ~1.4T tokens. GPT-3 (175B) was undertrained at 300B tokens.
Compute: C ≈ 6 · N · D (FLOPs)
Rough approximation: 6 FLOPs per parameter per token for a forward+backward pass.

In practice, modern models (LLaMA 3, Gemma) train way beyond Chinchilla-optimal because inference cost matters more: a smaller model trained on more data is cheap to serve. LLaMA 3 8B trains on 15T tokens (1875× parameter count).

Scaling laws let you predict the loss of a large run from much smaller ones — the GPT-4 report predicted its final loss from runs with up to 10,000× less compute. Run small models, fit the power law, extrapolate — this is how frontier labs plan training.
Interactive — scaling law curves

Python — fit scaling law#

Pattern bridge: Power-law relationships between compute, data, parameters, and loss. The same regression curves that describe natural phenomena. In markets, market cycles follow their own scaling laws — longer trends require proportionally more capitulation to reverse. The laws assume every token is worth training on; data curation is what makes that true.
Worked example

With 10²⁴ training FLOPs and Chinchilla’s rule of about 20 tokens per parameter, C = 6ND gives a model of 91B parameters trained on 1.8T tokens. Llama 3 8B was trained on about 15T tokens: 1,875 per parameter, 94× the Chinchilla ratio. Chinchilla optimises training compute; models that will be served billions of times are deliberately “over-trained” to be cheap to run.

Where it misleads
  • Scaling laws predict loss, not specific abilities; some skills appear abruptly or not at all at a given scale.
  • Fitted exponents depend on data, architecture and the range of sizes fitted; extrapolating far beyond it is a bet.
  • Accounting for inference changes the optimum (Sardana & Frankle 2024).
In code
C = 1e24                                        # training FLOPs budget
N = (C / (6 * 20)) ** 0.5                       # Chinchilla: C = 6ND with D = 20N
D = 20 * N
llama3_8b_ratio = 15e12 / 8e9                   # tokens per parameter: Llama 3 8B, ~15T tokens
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library
import numpy as np
from scipy.optimize import curve_fit

def power_law(x, a, b, c):
    return a * x**(-b) + c

# Example: fit loss vs parameters from small runs
params = np.array([1e6, 1e7, 1e8, 5e8, 1e9])
losses = np.array([4.2, 3.5, 2.9, 2.5, 2.3])

popt, _ = curve_fit(power_law, params, losses)
# Predict loss at 70B
predicted = power_law(70e9, *popt)
print(f"Predicted loss at 70B: {predicted:.3f}")
Sources
  • J. Kaplan et al., “Scaling Laws for Neural Language Models”, arXiv:2001.08361, 2020
  • J. Hoffmann et al., “Training Compute-Optimal Large Language Models”, NeurIPS, 2022
  • N. Sardana & J. Frankle, “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws”, ICML, 2024

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. What did the Chinchilla paper conclude about compute-optimal training?

Show answer

C. Hoffmann et al. (2022) found earlier large models were undertrained: a smaller model on more data matched them for the same compute.

2. Loss falls as a power law in compute. What does that imply?

Show answer

A. On a log-log plot the curve is a straight line: steady gains require exponentially growing budgets.

3. What do scaling laws predict reliably?

Show answer

B. Smooth loss curves can hide abrupt-looking jumps on individual tasks, which depend heavily on how the task is scored.

Linked from 1
← Previous
Mixture of Experts
Open in the full reader, with the topic sidebar