Chinchilla · Power Laws✓ Mathematical
◆ The PatternPredicting performance before spending millions on compute
Scaling laws reveal that LLM loss follows power laws in parameters (N), data (D), and compute (C). The Chinchilla paper showed that most models were trained on too little data — the optimal ratio is roughly 20 tokens per parameter.
L(N) ≈ (N_c / N)^α_N L(D) ≈ (D_c / D)^α_D
Loss decreases as a power law. α_N ≈ 0.076, α_D ≈ 0.095 from Kaplan et al.
Chinchilla optimal: D_opt ≈ 20 · N
For a 70B model, train on ~1.4T tokens. GPT-3 (175B) was undertrained at 300B tokens.
Compute: C ≈ 6 · N · D (FLOPs)
Rough approximation: 6 FLOPs per parameter per token for a forward+backward pass.
In practice, modern models (LLaMA 3, Gemma) train way beyond Chinchilla-optimal because inference cost matters more: a smaller model trained on more data is cheap to serve. LLaMA 3 8B trains on 15T tokens (1875× parameter count).
Scaling laws let you predict the loss of a $100M training run from a $1K experiment. Run small models, fit the power law, extrapolate — this is how frontier labs plan training.
Interactive — scaling law curves
Python — fit scaling law#
import numpy as np
from scipy.optimize import curve_fit
def power_law(x, a, b, c):
return a * x**(-b) + c
# Example: fit loss vs parameters from small runs
params = np.array([1e6, 1e7, 1e8, 5e8, 1e9])
losses = np.array([4.2, 3.5, 2.9, 2.5, 2.3])
popt, _ = curve_fit(power_law, params, losses)
# Predict loss at 70B
predicted = power_law(70e9, *popt)
print(f"Predicted loss at 70B: {predicted:.3f}")Pattern bridge: Power-law relationships between compute, data, parameters, and loss. The same regression curves that describe natural phenomena. In markets, market cycles follow their own scaling laws — longer trends require proportionally more capitulation to reverse.