Next-Token Prediction✓ Mathematical
◆ The PatternLearning language from trillions of tokens
Pre-training is the foundation: train a randomly-initialized transformer to predict the next token on a massive text corpus. The cross-entropy loss between predicted and actual next tokens drives all learning. The model develops grammar, facts, reasoning, and code — all from this single objective.
L = −(1/T) Σ log P(token_t | token_1, ..., token_{t−1})
Average negative log-likelihood over all positions. Lower loss = better predictions.
Perplexity = e^L = 2^(L/ln2)
Intuition: average number of "choices" the model is uncertain between. PPL of 10 ≈ choosing among 10 options.
Training recipe: AdamW optimizer (β₁=0.9, β₂=0.95), cosine learning rate schedule with warmup, weight decay 0.1, gradient clipping at 1.0, bf16 mixed precision, sequence packing, batch size ramp-up.
LLaMA 3 70B: 15T tokens, ~1e25 FLOPs, ~6000 GPU×months on H100s. The cost of pre-training a frontier model is $10M–$100M+ in compute alone.
Interactive — training loss curve
Python — pre-training loop skeleton#
import torch
from torch.nn import CrossEntropyLoss
optimizer = torch.optim.AdamW(model.parameters(),
lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
optimizer, T_max=total_steps)
for batch in dataloader:
input_ids = batch['input_ids'] # (B, T)
logits = model(input_ids[:, :-1]) # predict
loss = CrossEntropyLoss()(
logits.reshape(-1, vocab_size),
input_ids[:, 1:].reshape(-1) # targets shifted by 1
)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
scheduler.step()
optimizer.zero_grad()Pattern bridge: Learning general patterns from massive data before specialization. Like building a prior distribution from large samples.