LLM Engineering

A held-out loss of 2.0 nats per token is a perplexity of 7.39: as uncertain, on average, as choosing among seven equally likely tokens. Training an 8B model on 15T tokens takes about 6 × N × D = 7.2×10²³ FLOPs. At 40% of an H100’s peak that is about 506,000 GPU-hours — before failed runs, restarts and experiments.

• Perplexity is comparable only between models with the same tokenizer and evaluation text.
• Lower pre-training loss does not guarantee better behaviour after fine-tuning, though the two are usually related.
• Achieved utilisation is often well below 40%; the cost estimate is a floor.

loss = 2.0                                       # nats per token on held-out text
perplexity = np.exp(loss)
flops = 6 * 8e9 * 15e12                         # 8B parameters, 15T tokens
gpu_flops = 989e12 * 0.40                       # an H100's dense bf16 peak at 40% utilisation
gpu_hours = flops / gpu_flops / 3600

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

import torch
from torch.nn import CrossEntropyLoss

optimizer = torch.optim.AdamW(model.parameters(),
    lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=total_steps)

for batch in dataloader:
    input_ids = batch['input_ids']          # (B, T)
    logits = model(input_ids[:, :-1])       # predict
    loss = CrossEntropyLoss()(
        logits.reshape(-1, vocab_size),
        input_ids[:, 1:].reshape(-1)        # targets shifted by 1
    )
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()Copy