How LLMs Get Faster Without Changing Their Answers: A Companion Guide

Source: How LLMs Get Faster Without Changing Their Answers | Speculative Decoding by Jia-Bin Huang · used as the primary text for this companion

Jia-Bin Huang's lecture How LLMs Get Faster Without Changing Their Answers is a guided tour of one of the most useful tricks in modern LLM serving: speculative decoding. The setup is counterintuitive. A large model is slow precisely because it writes one token at a time, each token requiring a full forward pass. So you ask a small, cheap model to guess several tokens ahead, then let the big model score all of them in a single pass and keep the ones it agrees with. The output distribution is provably unchanged. The video then traces how the field has spent three years making that draft smarter: from a single linear guess, to draft trees, to heads bolted onto the target, to look-ahead learned during pre-training, to a drafter that emits a whole block of tokens in one parallel pass.

The problem: one token per forward pass

The video opens with the bottleneck. A transformer produces a probability distribution over the next token, you sample one, append it, and feed the whole sequence back in. The next token cannot be written until all preceding ones are available, so decoding is inherently serial. Each token costs one full forward pass through every layer.

There is one place where the model is already fast: when the tokens are known. During parallel prefill, the model scores every position in a prompt at once in a single pass, because nothing about position i depends on a token that has not yet been generated. Prefill is fast precisely because it is parallel. The question speculative decoding asks is blunt: if parallel scoring is cheap, can we get someone to hand us tokens to score for free?

The core trick: draft, then verify in parallel

Let a small draft model quickly guess K tokens ahead. Then run the large target model once. Because the target only has to score the proposed tokens, it can evaluate all K positions in a single forward pass — the same parallel scoring prefill already does. Walk the proposals left to right, stop at the first rejection, sample one correction token, and discard the rest.

The speedup is governed entirely by how many of the K drafts survive that single target pass. If the draft is good, one expensive pass buys several tokens. If the draft is bad, you still get the one token you would have produced anyway — you only wasted the cheap draft work. This is why the entire subsequent history of the field is a campaign to make the draft better without making it expensive.

Why the answers do not change: rejection sampling

The subtle part is that the draft model's distribution Q is not the target's distribution P. Naively accepting draft tokens would shift the output toward whatever the draft over-predicts. The fix is a modified rejection sampling rule that makes the final distribution identical to sampling from the target alone.

Accept a proposed token x with probability P(x) / Q(x), capped at 1. Where the draft already assigns more probability than the target (Q > P), the cap never binds and you accept at the ratio — thinning out the over-predicted tokens. Where the draft assigns less (Q < P), the ratio exceeds 1, so you cap at 1 and accept every occurrence — those tokens are rare in the draft, so keeping them all does not overshoot.

After a rejection you cannot just sample from P directly, because some mass has already been paid out to accepted drafts. You sample from the correction distribution: the target mass P(x) minus the overlap min(P, Q) that the draft already consumed, renormalized. Together, the acceptance rule and the correction distribution preserve the target's distribution exactly. This is the losslessness guarantee that runs through every method below — the draft can be aggressively wrong, and the output still matches the target.

The diagram shows the video's worked example with four tokens. The draft over-predicts cat and cow, so those are thinned by the ratio; it under-predicts dog and bear, so every occurrence is accepted. After a rejection, the leftover target mass (the part the accepted drafts did not cover) is renormalized into the correction distribution — here, 50% dog and 50% bear, exactly the 0.1 the draft was short on each.

The campaign to make the draft better

Once losslessness is guaranteed, the only lever is the accepted length — how many drafts survive before the first rejection. The rest of the video is a progression of ideas for raising it, each removing a constraint the previous method accepted.

Draft trees: verifying a tree in one pass

The first real upgrade is structural. Linear drafting throws away an entire pass on a single early rejection. But several independent guesses usually share a prefix, so they can be merged into a draft tree. The trick is verifying the whole tree in one forward pass. You flatten the tree into a single sequence, then replace standard causal attention with a tree-structured attention mask: each token attends only to the path from the root to itself, never to tokens in sibling branches. Without this, a token would leak information from an unrelated branch simply because that branch sits earlier in the flattened sequence. With it, every candidate path is preserved in the same flattened sequence, and the target commits the longest accepted prefix along any valid branch.

Medusa: draft from the target's own hidden states

A separate draft model is a deployment burden: it must share the target's tokenizer and vocabulary, stay in sync, and compete for the same GPU memory. Medusa removes the second backbone. It attaches several lightweight heads to the target's final hidden state, each predicting a different future position. The heads produce per-position distributions; you take a few top candidates from each and assemble a sparse tree, verified in one pass with tree attention — the token-tree verification paradigm introduced by SpecInfer (Miao et al., ASPLOS'24). A lossless variant stays exact via rejection sampling; a typical variant relaxes exactness to accept more drafts.

MTP: learn the look-ahead during pre-training

Medusa's heads are bolted on after pre-training. Multi-token prediction (MTP) asks why not bake look-ahead into the training objective from the start. A single hidden state branches into several parallel predictions, one per future position. The naive version makes those predictions independent, so each head guesses without seeing the others. DeepSeek-V3 fixes this by chaining its MTP modules: each extra token conditions on the previously drafted one, preserving a causal chain across prediction depths. Each module takes the target's intermediate state plus the embedding of the prior draft, normalizes and concatenates them, projects, and runs a transformer block. During training an auxiliary loss teaches look-ahead; at inference DeepSeek-V3 keeps a single module and the main model verifies the extra token in parallel. The reported payoff: an 85–90% acceptance rate for the second token across generation topics, and 1.8× tokens-per-second. The trade-off, as the video notes, is that the number of extra predictions is fixed at training time.

EAGLE: one recurrent feature-level drafter

MTP's look-ahead depth is fixed. EAGLE reuses one lightweight draft module in a loop — each predicted token feeds back in to produce the next, so a single module drafts as many positions as inference needs. The key move is drafting at the feature level rather than the token level: high-level feature sequences are easier to extrapolate than discrete tokens. Because the next feature is uncertain and depends on which token was actually sampled, EAGLE conditions the drafter on both the previous feature and the sampled token — the feature carries continuous state, the token resolves which discrete path was taken. A regression loss teaches the drafter to match the target's next feature.

EAGLE-2 fixes the static tree. EAGLE grows the same branching pattern every step regardless of context, wasting branches on near-certain tokens. EAGLE-2 makes the tree context-aware: expand where the next token is uncertain, stay narrow where it is obvious. The EAGLE-2 paper reports speedup ratios of 3.05–4.26×, which is 20–40% faster than EAGLE-1, with no extra training and no change to the target's output. EAGLE-3 goes further: it abandons feature prediction for direct token prediction and fuses lower, middle, and upper layer features via a technique called training-time test. Across the paper's evaluations EAGLE-3 reaches up to 6.5× the baseline speed, about 1.4× over EAGLE-2.

DFlash: draft the whole block in one parallel pass

Every method above still drafts autoregressively — one token at a time through the recurrent module, so draft cost grows linearly with the number of tokens and error accumulates. DFlash, from Z Lab and Inco AI, breaks that: it drafts an entire block in a single parallel pass. Anchor the last token the target accepted, put mask tokens in every slot you still need to guess, and run a small stack of bidirectional attention and MLP layers once. One pass yields a hidden state at every mask position, so you sample the whole block in one shot.

To keep the draft grounded, DFlash does not feed only the last token's features. It collects the target's layer features for every verified prefix position, turns them into keys and values, and injects them into each draft layer's KV cache — so the drafter sees the full prefix context, and the input does not fade as you go deeper into the stack. The target still verifies the block, so the output stays correct. DFlash reports over 6× lossless acceleration, up to 2.5× higher than EAGLE-3.

DFlash 2: a selector and local mixing

Parallel drafting is fast but locally incoherent: each position picks its token independently, so a block can be locally plausible yet globally nonsensical — "jump fast the same fox" is not a sentence, and two neighbors can even pick the same word. The right tokens are usually already on the short lists; the problem is selecting a coherent path through them.

DFlash 2 keeps the top 16 candidates at each position and scores every adjacent pair. The score has two terms: DFlash's own logit for the candidate (how much the drafter already liked it alone), plus a compatibility term — two learned linear maps embed the predecessor and the candidate into 256-dimensional vectors, matched under a context gate that decides which parts of the match count. In essence this is a low-rank bilinear attention over adjacent candidates. Scoring stays fully parallel; the only sequential work is a short greedy walk over precomputed scores from the last verified token. This is the path selector.

The selector fixes which candidate to pick, but it cannot invent a token that never made the list. The drafter's accuracy still decays toward the end of the block — the blog reports top-1 recall falling from 85.4% at position 0 to 72.9% at position 6, and top-16 recall from 99.5% to 87.8%. That is a backbone problem, not a selector problem. So DFlash 2 splits the job: attention keeps reading long-range context, and a small depthwise convolution handles local mixing inside the block. Before and after each attention and MLP layer, a two-tap conv mixes each position with its predecessor (reaching back to the last verified token at the first draft position), and the mixing weights can be made adaptive — a base kernel plus a correction from the current hidden state. All of it stays parallel. The video reports mean accepted length rising from DFlash's 4.92 tokens to 5.97.

Recap: what each step removed

The video closes by framing the whole arc as a sequence of removed constraints. Speculative decoding began as draft-and-verify. Then the field enhanced draft quality (trees), unified the draft and target models (Medusa), learned look-ahead during pre-training (MTP), made sequential forecasts of upcoming features (EAGLE), and finally enabled fully parallel drafting (DFlash). At every step the losslessness guarantee from rejection sampling is what makes the experimentation safe — you can make the draft arbitrarily aggressive and the output still matches the target.

Glossary

• Autoregressive decoding — generating one token at a time, each token requiring a full forward pass because it depends on all preceding tokens.
• Parallel prefill — scoring every position of a known sequence in one forward pass; fast because nothing depends on a not-yet-generated token.
• Draft / target model — the small efficient model that guesses ahead (draft) and the large powerful model that verifies (target).
• Rejection sampling — the acceptance rule min(1, P/Q) plus a correction distribution that makes the final output identical to sampling from the target alone.
• Accepted length — how many draft tokens survive a single verification pass; the single number governing speedup.
• Tree attention — a custom attention mask letting a flattened draft tree be verified in one pass, each token attending only to its ancestors.
• Logit — the raw score a model produces for a token before softmax turns it into a probability.
• LM head — the output projection plus softmax that converts a hidden state into a next-token distribution.
• Feature-level drafting — extrapolating continuous hidden-state features instead of discrete tokens, because feature sequences are smoother to predict.
• Block diffusion — DFlash's paradigm: predict an entire block of tokens in one parallel (non-causal) forward pass, conditioned on target features.

Read further

Related on the frontier