LLM Engineering

A decoder-only model scores a sequence as a product of next-token probabilities. Four tokens with probabilities 0.5, 0.3, 0.9 and 0.2 have a log-probability of −3.61: the sequence as a whole has probability 0.027, and long texts become astronomically unlikely. The causal mask lets each token see only what came before — 50% of the full attention pairs for 1,000 tokens.

• Low probability is not low quality: long, specific, correct answers are always improbable sequences.
• Left-to-right generation cannot revise what it already wrote, which is why planning and self-correction need extra steps.
• Encoder-decoder models remain competitive for some tasks (translation, structured outputs) at smaller scales.

p = np.array([0.5, 0.3, 0.9, 0.2])              # P(token_t | everything before it)
logp = np.log(p).sum()
seq_prob = np.exp(logp)
n = 1_000
pairs_causal, pairs_full = n * (n + 1) // 2, n * n   # attention pairs with and without the causal mask

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

import torch

def causal_mask(seq_len):
    """Lower-triangular mask for causal (left-to-right) attention"""
    mask = torch.triu(torch.ones(seq_len, seq_len), diagonal=1)
    return mask.masked_fill(mask == 1, float('-inf'))

# Usage in attention
scores = Q @ K.T / d_k**0.5
scores = scores + causal_mask(seq_len)  # mask future
weights = torch.softmax(scores, dim=-1)
output = weights @ VCopy