Draft & Verify✓ Mathematical
◆ The PatternUse a fast model to draft, a large model to verify — 2–3× speedup
Autoregressive generation is memory-bound: each token requires loading all model weights but does minimal computation. Speculative decoding uses a small draft model to generate K candidate tokens, then the large model verifies all K in one forward pass (which is compute-bound, so it's fast).
Draft: generate K tokens with small model M_s (fast)
The draft model should be ~10-20× smaller. E.g., 0.5B draft for 70B target.
Verify: run target model M_t on all K tokens in parallel → accept/reject each
Accept token i if P_target(token_i) ≥ P_draft(token_i). On rejection, resample from adjusted distribution.
The key guarantee: speculative decoding produces exactly the same distribution as standard generation — it's lossless. Speedup depends on draft model quality (acceptance rate). Typical: 60–80% acceptance → 2–3× throughput.
Medusa and Eagle add extra heads to the model itself instead of using a separate draft model — eliminating the need for draft-target distribution matching.
Interactive — speculative decoding timeline
Python — speculative decoding loop#
def speculative_decode(target, draft, prompt, K=4):
tokens = prompt.clone()
while len(tokens) < max_len:
# 1. Draft K tokens
draft_tokens, draft_probs = [], []
for _ in range(K):
logits = draft(tokens)
p = logits[:, -1].softmax(-1)
t = torch.multinomial(p, 1)
draft_tokens.append(t)
draft_probs.append(p[0, t.item()])
tokens = torch.cat([tokens, t], dim=-1)
# 2. Verify all K with target (single forward pass)
target_logits = target(tokens)
for i in range(K):
pos = len(tokens) - K + i
p_target = target_logits[:, pos-1].softmax(-1)
p_t = p_target[0, draft_tokens[i].item()]
# Accept with min(1, p_target/p_draft)
if torch.rand(1) < (p_t / draft_probs[i]):
continue # accepted
else:
# Reject: resample, discard rest
tokens = tokens[:, :pos]
break
return tokensPattern bridge: A small model drafts, a large model verifies — the same principle as generator/discriminator in GANs. In markets, contrarian thinking verifies what the crowd drafts.