LLM Engineering

Eight candidate tokens. At temperature 0.7 the distribution has 1.69 bits of entropy and top-p = 0.9 keeps 3 tokens; at 1.0, 2.07 bits and 4 tokens; at 1.3, 2.33 bits and 5. Temperature changes how flat the distribution is, and nucleus sampling adapts how many tokens survive to that shape.

• High temperature adds diversity and errors together; for factual or code tasks, low temperature is usually better.
• Fixed top-k ignores the shape of the distribution; top-p and min-p adapt, but all are heuristics tuned by eye.
• Sampling settings interact: temperature applied before or after truncation gives different results.

logits = np.array([3.0, 2.5, 2.0, 1.0, 0.5, 0.0, -1.0, -2.0])

def probs(T):
    z = np.exp((logits - logits.max()) / T); return z / z.sum()

def nucleus_size(p, top_p=0.9):
    return int(np.searchsorted(np.cumsum(np.sort(p)[::-1]), top_p) + 1)

entropy = lambda p: -(p * np.log2(p)).sum()
table = {T: (round(entropy(probs(T)), 2), nucleus_size(probs(T))) for T in (0.7, 1.0, 1.3)}

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

def sample(logits, temperature=0.8, top_p=0.9, top_k=50):
    logits = logits / temperature
    # Top-K filtering
    if top_k > 0:
        indices_to_remove = logits < logits.topk(top_k).values[..., -1:]
        logits[indices_to_remove] = float('-inf')
    # Top-P (nucleus) filtering
    sorted_logits, sorted_idx = logits.sort(descending=True)
    cumsum = sorted_logits.softmax(dim=-1).cumsum(dim=-1)
    remove = cumsum - sorted_logits.softmax(dim=-1) >= top_p
    sorted_logits[remove] = float('-inf')
    logits.scatter_(-1, sorted_idx, sorted_logits)
    probs = logits.softmax(dim=-1)
    return torch.multinomial(probs, 1)Copy