21 — Inference

Speculative Decoding#

Draft & VerifyMathematical 1 min read
◆ The PatternUse a fast model to draft, a large model to verify — 2–3× speedup

Autoregressive generation is memory-bound: each token requires loading all model weights but does minimal computation. Speculative decoding uses a small draft model to generate K candidate tokens, then the large model verifies all K in one forward pass (which is compute-bound, so it's fast).

Draft: generate K tokens with small model M_s (fast)
The draft model is usually much smaller — 10× to 100× — e.g. a 1B draft for a 70B target.
Verify: run target model M_t on all K tokens in parallel → accept/reject each
Accept token i if P_target(token_i) ≥ P_draft(token_i). On rejection, resample from adjusted distribution.

The key guarantee: speculative decoding produces exactly the same distribution as standard generation — it's lossless. Speedup depends on draft model quality (acceptance rate). Leviathan et al. report 2–3× faster generation.

Medusa and Eagle add extra heads to the model itself instead of using a separate draft model — eliminating the need for draft-target distribution matching.
Interactive — speculative decoding timeline

Python — speculative decoding loop#

Pattern bridge: A small model drafts, a large model verifies — the same principle as generator/discriminator in GANs. In markets, contrarian thinking verifies what the crowd drafts.
Worked example

A draft model proposes 4 tokens and the large model checks them in one pass. If each draft token is accepted with probability α, the expected tokens per large-model pass is (1 − α⁵)/(1 − α): 2.31 at α = 0.6, 3.36 at 0.8 and 4.10 at 0.9. The output distribution is exactly the large model’s; only the speed changes.

Where it misleads
  • The speed-up is in passes, not wall-clock; the draft model’s own cost and a larger batch can eat it.
  • Acceptance rates vary by task: code and boilerplate accept well, creative text poorly.
  • Under heavy batching the large model is already compute-bound, and speculation helps much less.
In code
def tokens_per_pass(alpha, gamma=4):            # Leviathan et al.: expected tokens per target-model pass
    return (1 - alpha ** (gamma + 1)) / (1 - alpha)

table = {a: round(tokens_per_pass(a), 2) for a in (0.6, 0.8, 0.9)}
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library
def speculative_decode(target, draft, prompt, K=4):
    tokens = prompt.clone()
    while len(tokens) < max_len:
        # 1. Draft K tokens
        draft_tokens, draft_probs = [], []
        for _ in range(K):
            logits = draft(tokens)
            p = logits[:, -1].softmax(-1)
            t = torch.multinomial(p, 1)
            draft_tokens.append(t)
            draft_probs.append(p[0, t.item()])
            tokens = torch.cat([tokens, t], dim=-1)

        # 2. Verify all K with target (single forward pass)
        target_logits = target(tokens)
        for i in range(K):
            pos = len(tokens) - K + i
            p_target = target_logits[:, pos-1].softmax(-1)
            p_t = p_target[0, draft_tokens[i].item()]
            # Accept with min(1, p_target/p_draft)
            if torch.rand(1) < (p_t / draft_probs[i]):
                continue  # accepted
            else:
                # Reject: resample, discard rest
                tokens = tokens[:, :pos]
                break
    return tokens
Sources
  • Y. Leviathan, M. Kalman & Y. Matias, “Fast Inference from Transformers via Speculative Decoding”, ICML, 2023
  • C. Chen et al., “Accelerating Large Language Model Decoding with Speculative Sampling”, arXiv:2302.01318, 2023

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 1
← Previous
Sampling
Open in the full reader, with the topic sidebar