Temperature · Top-K · Top-P✓ Mathematical
◆ The PatternControlling creativity by shaping the probability distribution
Instead of argmax, sample from the distribution — but shape it first. Temperature sharpens or flattens. Top-K limits to K most likely tokens. Top-P (nucleus) limits to the smallest set that sums to P probability.
Temperature: P'(token) = softmax(logit / T)
T<1 → sharper (more confident). T>1 → flatter (more random). T→0 = greedy.
Top-K: zero out all but top K logits, renormalize
K=50 is common. Prevents sampling very unlikely tokens (garbage).
Top-P (nucleus): keep tokens until cumulative P ≥ p, zero rest
Adaptive: for confident predictions, keeps few tokens. For uncertain, keeps many. p=0.9–0.95 typical.
In practice, combine them: temperature + top-P is the standard. Repetition penalty divides logits of recently-generated tokens by a factor (1.1–1.3) to reduce loops. Min-P is a newer approach that scales the threshold with the top token's probability.
Temperature 0.0–0.3 for code/math (deterministic). Temperature 0.7–1.0 for creative writing. Top-P 0.9 is a solid default for most tasks.
Interactive — probability distribution shaping
Python — sampling with temperature + top-p#
def sample(logits, temperature=0.8, top_p=0.9, top_k=50):
logits = logits / temperature
# Top-K filtering
if top_k > 0:
indices_to_remove = logits < logits.topk(top_k).values[..., -1:]
logits[indices_to_remove] = float('-inf')
# Top-P (nucleus) filtering
sorted_logits, sorted_idx = logits.sort(descending=True)
cumsum = sorted_logits.softmax(dim=-1).cumsum(dim=-1)
remove = cumsum - sorted_logits.softmax(dim=-1) >= top_p
sorted_logits[remove] = float('-inf')
logits.scatter_(-1, sorted_idx, sorted_logits)
probs = logits.softmax(dim=-1)
return torch.multinomial(probs, 1)Pattern bridge: Temperature, top-k, top-p control the randomness of generation. Temperature is softmax temperature scaling. In statistics, sampling from distributions is the foundation.