Speculative Decoding: Draft with a Small Model, Verify in Parallel, but T5 Speedups Do Not Represent Every Serving Stack

Pair this with the three-pass approach. In this site’s foundations sequence, the note follows InstructGPT. InstructGPT covers post-pretraining alignment; Speculative Decoding keeps target weights frozen and accelerates inference through drafting plus parallel verification, using wall-clock speedup and acceptance rate alpha as its core evidence.

YOLO also treats latency as a first-class metric, but the experiments are not interchangeable: this paper measures T5-XXL decoding, not VOC mAP.

The paper in 90 seconds

• Problem: Decoding K tokens from a large autoregressive Transformer takes K serial forward passes; each step is often limited by memory bandwidth rather than raw arithmetic throughput, leaving spare parallel capacity idle (Section 1).
• Core insight: Speculative Decoding lets a small model M_q autoregressively draft gamma tokens, then runs the target model M_p in parallel on the prefix through each draft position to obtain distributions p_1 through p_{gamma+1}. Speculative sampling (accept/reject testing plus a residual-distribution correction) decides how many drafts to accept: after the first rejection it samples a correction from a residual distribution; if all pass, it samples an extra token from the next target distribution. The control point is lossless parallel verification versus step-by-step target-only decoding; the output distribution is identical to M_p alone (Algorithm 1, Appendix A.1).
• Strongest evidence: T5-XXL 11B as M_p and off-the-shelf T5-small 77M as M_q versus the T5X baseline, batch=1, single TPU-v4 (Table 2): WMT EnDe 3.4X (temp=0, gamma=7, alpha=0.75) and 2.6X (temp=1, alpha=0.62); CNN/DM 3.1X and 2.3X. The abstract and Section 4 also report a 2X-3X band relative to T5X.
• Main boundary: You need a task-aligned draft model and hardware that can evaluate the draft positions concurrently; total arithmetic operations can rise (Sections 3.4 and 6). This is a 2023 Google T5X experimental contract, not a vLLM or TensorRT-LLM product SLA, not GPTQ bitwidth, and not Medusa or EAGLE draft heads. InstructGPT 85±3% win rates, Transformer WMT BLEU, and YOLO mAP are not in this PDF.

Core engineering takeaway: Speculative Decoding’s lasting contribution is using a draft model to reduce wall-clock decoding time without changing the target distribution. Table 2’s 3.4X applies to the paper’s setup and cannot guarantee performance for every 2026 LLM serving stack.

Huahua’s one-liner

The small model guesses, the large model verifies in one parallel pass; accept drafts through the speculative-sampling test, otherwise correct back to M_p. But Table 2’s 3.4X is T5X on a TPU-v4, not vLLM dashboard tokens/s.

Version and reading scope

This article reads Leviathan et al., ICML 2023 as arXiv:2211.17192 v2 (revised 2023-05-18). The arXiv v2 record lists CC BY 4.0. Author order follows v2: Yaniv Leviathan, Matan Kalman, and Yossi Matias (Leviathan and Kalman contributed equally).

Beyond the abstract, this reading examines Section 2 (algorithm and speculative sampling), Section 3 (acceptance rate and wall-clock analysis: Theorem 3.8, Figures 2-5, Table 1), Section 4 T5-XXL experiments (Tables 2-3), Section 6 limitations, and paper readability as of 2026-08-28. GPTQ, AWQ, FlashAttention TFLOPS, vLLM, Medusa, EAGLE, and Lookahead numbers are not written back.

The question a reader should answer

When you already have a trained large autoregressive model M_p and decoding latency is the product bottleneck, should you keep sampling M_p one token at a time, or let a smaller M_q draft and have M_p verify in parallel? Leviathan et al. choose the latter and report both a distribution-equivalence proof and T5-XXL wall-clock tables.

The precise read is not “this is the fastest 2026 LLM inference stack.” The real question is: how speculative sampling preserves the target distribution, how alpha and gamma translate into tokens per target call, what Table 2 actually contracts for hardware, and which later serving, quantization, or learned-draft numbers must not be copied into this paper’s tables.

Evidence map

Layer	How this article uses it
Paper directly supports	Figure 1 unconditional-generation sketch (green=accepted drafts, red=rejected, blue=corrected); Algorithm 1; Equation (1) expected tokens per iteration; Theorem 3.5 beta=1-D_LK(p,q) and Corollary 3.6 alpha=E(min(p,q)); Theorem 3.8 wall-clock formula; Table 1 theoretical speed/ops; Table 2 T5-XXL measurements; Table 3 alpha across tasks; Figure 5 encoder-decoder trace.
Author claims	Large-model decoding can speed up via speculative execution without changing the output distribution; extra parallelism pays when memory bandwidth is the bottleneck; off-the-shelf small Transformers as M_q already yield 2X-3X; negligible-cost drafts such as n-grams can still have nonzero alpha.
Not established	vLLM or TensorRT-LLM SLAs on arbitrary hardware; GPTQ or AWQ quality; Medusa or EAGLE learned draft heads; FlashAttention kernel wins; adaptive computation with identical distribution guarantees after retraining.
Bloss0m engineering judgment	Place this note in the lossless-inference-efficiency section, after InstructGPT. YOLO offers a useful latency comparison, while InstructGPT helps distinguish model architecture from later procedures. GPTQ WikiText, vLLM tokens/s, and Medusa acceptance do not belong in Table 2.

Why the previous approach is insufficient

Sections 1 and 5 set the context. Standard autoregressive decoding calls M_p once per emitted token, so K tokens need K serial forwards. Even when per-step FLOPs are modest, weight and KV-cache memory traffic often dominates latency.

Common acceleration routes differ from this paper:

• Distillation, quantization, architecture changes (Hinton et al., Hubara et al., So et al.): usually change the model or retrain and do not guarantee the same distribution as M_p.
• Adaptive computation and early exits (Han et al., Schwartz et al.): heuristics skip work but do not guarantee M_p’s distribution.
• Blockwise Parallel Decoding and SAD (Stern et al., Sun et al.): need extra training or restrict to greedy or copy-input settings, not general stochastic lossless decoding.
• Greedy or nucleus sampling with M_p alone: distribution-correct but cannot verify multiple tokens in one M_p call.

InstructGPT addresses alignment training; Transformer addresses sequence-transduction architecture. Neither solves decoding wall-clock without changing weights.

Core intuition

Before Algorithm 1, picture target model M_p continuing an English summary.

Standard decoding: M_p runs one step, samples t_1, runs again, samples t_2, and so on. Each step waits for a full M_p forward.

Speculative Decoding: a much cheaper M_q (for example T5-small) autoregressively guesses gamma tokens x_1 through x_gamma. M_p evaluates the prefix and each prefix plus drafts in parallel, yielding p_1 through p_{gamma+1}. Each x_i faces a rejection test with acceptance probability min(1, p_i(x_i)/q_i(x_i)). At the first rejection, sample a correction from p’=norm(max(0,p-q)). The residual p’ conditional on rejection is not p itself: combining the accepted and rejected branches recovers the p marginal (Appendix A.1). If all drafts pass, sample one more token from p_{gamma+1}.

Three easy confusions:

• Target-only stepwise decoding (baseline): correct distribution, wall-clock equals K serial M_p calls.
• Speculative Decoding (this paper): still correct (Appendix A.1), wall-clock depends on how many drafts survive each round.
• Deploy a distilled small model alone: may be faster, but no longer M_p’s distribution; a different problem statement.

Walk one example through the method

The following walks one Algorithm 1 round on a simplified T5 English-to-German fragment (a Bloss0m teaching example, not a Table 2 row).

1. Input: the encoder has consumed the English sentence; the decoder prefix is <s> Die (partial German already emitted).
2. Intermediate representation: M_q=T5-small autoregressively drafts gamma=3 tokens, for example Katze / schläft / gut, with per-step q_i recorded.
3. Model or system decision: M_p=T5-XXL computes p_1 through p_4 in parallel. Suppose Katze and schläft pass rejection (r_i <= p_i(x_i)/q_i(x_i)) but gut fails; sample correction friedlich from p’=norm(max(0,p_3-q_3)).
4. Output: new prefix <s> Die Katze schläft friedlich (two accepted drafts plus one corrected token this round).
5. Likely failure point: if M_q is a poor match to M_p (low alpha), almost every round emits only one token while still paying gamma M_q steps, slower than baseline (Corollary 3.9: under the analysis assumptions, alpha>c guarantees that some gamma gives a speedup, not that every gamma does). A mismatch in task distributions can also lower alpha. Algorithm 1 requires p and q on the same token vocabulary: probabilities from different tokenizers cannot be inserted directly into the acceptance and residual formulas without alignment (Section 2, Appendix A.1). Identical architectures are not required.

Technical mechanism

Speculative sampling (Section 2.3)

To sample from p(x), propose x from q(x): accept when q(x)<=p(x); when q(x)>p(x), reject with probability 1-p(x)/q(x) and resample from p’(x)=norm(max(0,p(x)-q(x))). Appendix A.1 shows the marginal remains p(x).

Algorithm 1: one decoding round (Sections 2.1-2.3)

• Inputs: M_p, M_q, prefix.
• M_q autoregressively produces x_{1..gamma} and q_i.
• M_p computes p_1 through p_{gamma+1} in parallel.
• n <- index of first failed rejection minus one (or gamma if all pass).
• If n<gamma, sample correction t from adjusted p_{n+1}; else sample t from p_{gamma+1}.
• Output: prefix plus accepted x_{1..n} plus t (at least one new token, with the complete sampling procedure preserving the target distribution).

Acceptance rate and expected throughput (Sections 3.1-3.2)

Acceptance rate beta_{x_{<t}} = Pr[accept x_t ~ q]; alpha = E(beta). Under an i.i.d. simplification, expected tokens per round:

$$E(\#\ \text{generated tokens}) = \frac{1-\alpha^{\gamma+1}}{1-\alpha}$$

Theorem 3.5 gives $\beta=\sum_x\min(p(x),q(x))=1-D_{LK}(p,q)$ for a fixed prefix; $D_{LK}$ here is total variation distance, not KL divergence. Corollary 3.6 averages over prefixes to obtain $\alpha=E[\beta]$: closer draft and target distributions generally yield higher acceptance.

Wall-clock speedup (Theorem 3.8, Section 3.3)

Let c be the ratio of one M_q step time to one M_p step time. Assume i.i.d. acceptance rates, enough parallel compute that multi-position verification takes one target-step wall-clock time, and sufficiently long generations. Each Algorithm 1 round costs about T(c*gamma + 1) and produces the expectation above, giving improvement factor:

$$\frac{1-\alpha^{\gamma+1}}{(1-\alpha)(\gamma c + 1)}$$

Section 3.3 discusses cheap drafting, but c must be measured for the model and implementation. Appendix A.3 / Table 4 reports c=0.02 for T5-small, c=0.04 for T5-base, and c=0.11 for T5-large with T5-XXL; c<0.05 therefore does not cover every draft model.

Figure 1, Section 1: speculative decoding sketch (green=accepted, red=rejected, blue=corrected). Source: arXiv PDF Figure 1. Cropped from the arXiv v2 PDF with the complete illustration and original caption. Original work by Leviathan, Kalman, and Matias, CC BY 4.0; only other page content was cropped away.

Figure 2, Section 3.1: E(tokens) versus alpha (Equation 1). Source: arXiv PDF Figure 2. License note as for Figure 1.

How to read the evidence

Table 2: T5-XXL wall-clock (Section 4.1)

Question: versus the T5X baseline, can off-the-shelf small T5 models as M_q deliver 2X-3X speedup with identical outputs? Controls: M_p=T5-XXL 11B; tasks WMT EnDe and CNN/DM summarization; batch=1; single TPU-v4; existing checkpoints; argmax (temp=0) and standard sampling (temp=1). Observation: T5-small (77M) wins overall: EnDe 3.4X (gamma=7, alpha=0.75) and 2.6X (gamma=7, alpha=0.62); CNN/DM 3.1X (gamma=5, alpha=0.65) and 2.3X (gamma=5, alpha=0.53). T5-large shows higher alpha but larger c, so speedup falls to 1.4X-2.2X across the two tasks and temperatures. Boundary: Google-internal T5X implementation; not default open-source vLLM; LaMDA 137B reports alpha only (Table 3), not Table 2 wall-clock.

Table 2, Section 4.1: T5-XXL empirical walltime. Source: arXiv PDF Table 2. The header, all twelve rows, and original table caption are retained. License note as for Figure 1.

Table 3 and the small-model experiment in Figure 1

Question: how does alpha move with M_q size and temperature? Observation: GPT-like 97M M_p plus 6M M_q on lm1b reaches alpha about 0.88-0.89 (Table 3); the Figure 1 sentence needs only 9 serial M_p calls for 38 tokens. Boundary: lm1b unconditional generation is not a production chat SLA.

Figure 5: encoder-decoder trace (Section 3.5)

Question: how does gamma reshape wall-clock structure? Observation: versus standard decoding (one long purple M_p block), gamma=3 or 7 trades multiple blue M_q segments for fewer purple M_p segments. Boundary: schematic, not a benchmark table.

Figure 5, Section 3.5: simplified trace diagram. Source: arXiv PDF Figure 5. The complete trace, legend, and original caption are retained. License note as for Figure 1.

Table 1: theoretical speed versus ops (Section 3.5)

Question: what does high alpha with large gamma cost? Observation: for example alpha=0.9 and gamma=10 yields theoretical speed 6.86X but ops 1.60X. Boundary: assumes c=hat{c}=0; plug into Theorem 3.8 for real hardware.

Ablations and design choices

• M_q size (Tables 2-3): alpha rises with larger M_q, but so does c; T5-small balances best on T5-XXL.
• gamma (Figure 3, Section 3.5): optimize numerically from alpha and c; higher alpha permits larger gamma.
• Sampling temperature (Table 2): temp=0 (argmax) beats temp=1 on both alpha and speedup because sharper distributions agree more often.
• Negligible-cost drafts (Section 3.6): an EnDe bigram M_q still has alpha about 0.2, worth about 1.25X at gamma=3, low but nonzero.

Limitations and threats to validity

1. Compute prerequisite: you must parallelize gamma+1 M_p forwards; if already saturated, the method does not help (Section 6).
2. Total operations: low alpha wastes parallel M_p work and M_q drafting (Theorem 3.11).
3. Draft quality: M_q must operate in a compatible token space and approximate the target task distribution. Identical architectures are not required for correctness; the paper mainly tests same-family models, and cross-modal or cross-task benefits remain unverified.
4. Hardware era: single TPU-v4, T5X; remeasure on 2026 GPU clusters.
5. Do not mix in later results: vLLM, TensorRT-LLM, GPTQ, Medusa, EAGLE, and FlashAttention benchmarks are outside this PDF.
6. Keep alignment and CV separate: InstructGPT win rates, Transformer WMT BLEU, and YOLO mAP must not enter Table 2.

Bloss0m engineering judgment and when not to use it

When to borrow this paper: when (a) you must preserve M_p’s exact sampling distribution, (b) decoding is memory-bandwidth bound with spare compute, (c) you have a same-family small checkpoint as M_q, and latency is a product metric (echoing YOLO putting cost on the same table).

When not to copy it blindly:

• You can accept an approximate distribution (distillation, quantization) and care more about memory footprint.
• You lack a suitable M_q or alpha is estimated too low.
• GPUs are already saturated and cannot host parallel M_p paths.
• You write 3.4X EnDe into a 2026 arbitrary LLM API p99 latency SLA.
• You confuse Medusa or EAGLE learned drafts with this paper’s speculative-sampling lossless contract.

Huahua’s judgment

From YOLO, keep “latency is a first-class metric.” From InstructGPT, keep “distinguish architecture from the training procedure.” From Speculative Decoding, add one more line: losslessness comes from speculative sampling; T5X 2X-3X is a 2023 experimental contract, not a 2026 serving product warranty.

Artifacts and reproducibility

As of 2026-10-30:

• Paper: arXiv abs, PDF v2, and the ICML 2023 proceedings page are readable.
• Code: the paper implements its experiments in T5X (Section 4.1), but provides no download link for a standalone experiment package. Public T5X is a training and inference framework; its availability alone does not establish a complete Table 2 reproduction setup.
• Models: the official T5X model catalog lists T5 v1.1 checkpoints. Pretrained weights are not the complete set of downstream task-fine-tuned checkpoints used in the paper; T5-XXL 11B still needs corresponding resources.

The smallest useful reproduction: implement one Algorithm 1 round on small GPT or T5 models and measure the distribution of accepted length n plus whether parallel M_p forwards are feasible. That validates mechanism, not 3.4X.

Three things to remember

1. Technical idea: M_q drafts, M_p verifies in parallel, and speculative sampling keeps the same distribution as M_p; the control point is lossless inference acceleration, not a new architecture.
2. Evidence: Table 2: T5-XXL plus T5-small, EnDe 3.4X/2.6X, CNN/DM 3.1X/2.3X; Figure 2 and Theorem 3.8 explain the alpha, gamma, and c tradeoff.
3. Boundary: The method needs a draft model plus parallel compute and is not GPTQ, vLLM, or Medusa. In the foundations sequence, this note’s specific topic is inference efficiency.

Further reading

If you have not read the preceding sequence, return to Transformer and InstructGPT. For reading method, see the three-pass approach. To compare latency on the evidence table with a CV example, read YOLO. GPTQ, FlashAttention, vLLM, Medusa, and EAGLE are intentionally not expanded here.

Primary sources

• Leviathan et al., “Fast Inference from Transformers via Speculative Decoding,” ICML 2023 / arXiv:2211.17192 v2
• ICML 2023 proceedings entry
• DOI 10.48550/arXiv.2211.17192