Inside Speculative Decoding: How AI Models Generate Faster Without Changing the Answer

Large language models generate text through an awkwardly sequential process. They predict one token, append it to the context, then repeat. Each step may mobilize billions of parameters merely to produce a small unit of text. Faster hardware helps, but it does not remove this dependency: token ten ordinarily cannot be finalized before token nine exists.

Speculative decoding approaches the constraint sideways. A cheaper model drafts several possible tokens in advance. The primary model then examines the batch and accepts as much of it as the decoding rules permit. When the draft is good, one expensive verification pass advances generation by several tokens rather than one.

The important claim is not that the smaller model replaces the larger one. It is that prediction can be separated from authority. One system proposes; another decides.

The sequential bottleneck beneath generation

During autoregressive inference, a transformer processes the prompt and stores intermediate attention data in a key-value cache. For each subsequent token, it reads that cache, computes a probability distribution over the vocabulary, selects a token, updates the cache, and begins again.

Caching avoids recomputing the entire history, but decoding remains iterative. It is also frequently constrained by moving model weights and cached data through memory rather than by arithmetic alone. A single-token step may leave available parallel compute underused, especially at small batch sizes.

Ordinary batching improves utilization by serving multiple sequences together. Yet batching can introduce queueing and does little for a solitary interactive request that needs a low time-to-next-token. Speculative decoding instead creates parallel work within one sequence: multiple drafted positions can be checked together.

The draft-and-verify mechanism

Consider a primary model that would normally choose the continuation “through the narrow gate.” A smaller draft model receives the same accepted context and proposes four tokens: “through the narrow door.” The primary model evaluates those proposed positions in one forward pass.

1. The draft model generates a short candidate sequence autoregressively.
2. The primary model scores each candidate token using the context that precedes that position.
3. A verification rule accepts a valid prefix of the draft.
4. At the first rejection, the system samples or selects a replacement from the primary model’s distribution.
5. Both models resume from the newly accepted context.

If the primary model validates “through the narrow” but rejects “door,” the system advances several positions before returning control to drafting. If disagreement occurs on the first token, the draft work has yielded little benefit.

The verifier can score several positions simultaneously because the entire proposed sequence is available. A causal attention mask still prevents each position from seeing future tokens, preserving the same directional structure used in normal generation.

Why exactness is possible

A tempting but incomplete implementation would accept a draft token whenever the primary model also ranks it first. That may work for greedy decoding, but sampled generation is different. Replacing samples from the primary distribution with convenient draft samples can alter the output distribution.

Exact speculative sampling corrects for this. Let the draft model assign probability q to a proposed token and the primary model assign probability p. The token is accepted with probability based on the ratio between those probabilities, capped at one. A token the primary model considers at least as plausible as the draft does can be accepted directly. When a token is rejected, a corrected residual distribution determines the replacement.

This accept-reject construction preserves the primary model’s target distribution under the assumed decoding procedure. The generated text is not guaranteed to match a separate conventional run token for token, because sampling itself is random. The stronger statement is that outputs follow the same distribution.

That guarantee has boundaries. Production systems often add top-k filtering, nucleus sampling, temperature changes, repetition penalties, grammar constraints, or custom logits processors. Each transformation must be represented consistently in verification. Otherwise the implementation may be fast while quietly changing behavior.

The economics of acceptance

The decisive variable is not merely how quickly the draft model runs. It is how many proposed tokens survive verification relative to the cost of producing and checking them.

Imagine two configurations. One drafts eight tokens cheaply but regularly disagrees at the second position. The other drafts four tokens at greater cost and commonly secures all four. The second may win because useful accepted tokens, not raw draft throughput, determine progress.

The optimal block length can also change during one response. Familiar syntax may support a long draft. A code identifier, uncommon name, or abrupt reasoning turn may require shorter speculation. This makes adaptive policies attractive: observe recent acceptance, then adjust how aggressively the system drafts.

What the hardware actually experiences

Algorithmic speedup does not translate automatically into lower latency. The draft and primary models consume memory, cache capacity, scheduling attention, and communication bandwidth. If both reside on the same accelerator, the draft may compete with the verifier. If they occupy separate devices, transferring state or coordinating steps can erase part of the gain.

Verification is valuable because scoring a short sequence can use accelerator parallelism more effectively than repeated single-token calls. But the result depends on model architecture, sequence length, batch size, precision, kernels, and memory layout. A deployment already running large continuous batches may gain less than a lightly batched conversational service.

There is also a distinction between latency and throughput. Speculation may make one request finish sooner while consuming additional total computation. Under heavy load, that extra work can reduce the number of requests served. A credible evaluation therefore needs at least three views:

• User latency: time to first token, time between visible tokens, and total completion time.
• System throughput: completed or accepted tokens per unit of infrastructure time.
• Resource cost: accelerator occupancy, memory use, energy, and cache pressure.

A single “tokens per second” figure can conceal whether the system became genuinely more efficient or merely spent more resources to make one stream appear faster.

Drafting without a separate model

The draft need not come from an independently trained miniature model. Some architectures attach early-exit or auxiliary prediction heads to intermediate layers of the primary network. Others use a compact component trained to predict several future tokens from internal representations. Tree-based variants propose multiple branches, allowing the verifier to recover useful progress even when one path fails.

These designs trade modularity for tighter integration. A separate draft model can be swapped, quantized, or routed by domain. An integrated drafter may reuse representations and avoid duplicating some memory, but it generally requires architectural changes or specialized training.

Simple pattern matching can also act as a proposal mechanism in constrained settings. Repeated text, retrieved passages, or predictable formatting may supply candidate spans. The unifying principle remains unchanged: obtain cheap guesses, then preserve the primary model as the authority.

Where the technique breaks down

Speculative decoding does not improve the model’s reasoning, factuality, or context capacity. It accelerates the path to whatever the primary model would have produced under the same decoding rules. A weak answer arrives sooner.

It is also a poor fit when generation is not the dominant cost. Retrieval, tool execution, network calls, prompt processing, and safety checks may occupy most of the request. Accelerating token emission then changes little about end-to-end latency.

Tokenizer mismatch creates another obstacle. Draft and verifier must agree about how candidate text maps to token positions, or the system needs a more complex reconciliation layer. Vocabulary differences can make seemingly identical text awkward to verify.

Finally, acceptance quality can drift. A draft model that matches ordinary conversation may perform badly on a new programming language, legal template, or multilingual workload. Aggregate benchmarks can hide these domain-specific failures.

The open frontier: speculation as a control problem

The deeper opportunity lies in deciding when, how, and with what to speculate. A production controller could choose among several drafters, vary proposal length, disable speculation under load, or route predictable spans to specialized mechanisms. Its signals might include recent acceptance depth, token entropy, request domain, available memory, queue pressure, and latency targets.

This introduces unresolved questions. Can acceptance be predicted reliably before paying for a draft? How should a scheduler balance one request’s latency against fleet-wide throughput? Can draft models be updated without destabilizing exactness or cache behavior? Which metrics reveal distributional changes introduced by custom decoding constraints?

Speculative decoding reveals a broader architectural pattern. Expensive intelligence does not need to perform every step of a task. A cheaper process can explore likely moves, provided a trusted process validates them with rigor. The durable innovation is therefore not merely faster text generation. It is a division of labor between conjecture and authority—and a new control layer deciding how much conjecture the system can afford.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.