Skip to content
All guidesCost & efficiency

Speculative decoding, how small models speed up big ones

How AI drafts and checks several text pieces together, why it can reduce waiting time, and when the extra work makes generation slower.

5 min read

A large language model normally generates an answer one token at a time. A token is a piece of text, which can be a word, part of a word or punctuation. Each new token depends on the text already produced. That makes a long answer a sequence of model runs that the reader waits through.

Speculative decoding can shorten that wait. A lightweight model proposes several tokens ahead, then the larger model checks those proposed positions together. When enough guesses are accepted, the system moves further through the answer with fewer sequential runs of the larger model.

Why checking several guesses can be faster

The smaller model is called the draft model. The larger one is the target model, whose behaviour the system aims to preserve. A pass means one run through a model. Drafting is cheap only relative to the target and the hardware serving it; being smaller does not guarantee a useful speed-up.

When a draft is available, the target already has the proposed preceding text for each position. It can calculate scores for those positions in parallel. It still respects the sequence of the text. It is doing several conditional calculations together because the guesses supply the missing inputs.

This is useful when a target model spends substantial time moving its stored parameters through memory and has spare capacity for calculation. Checking a short block can cost less time than making a separate target pass for every token. The draft itself adds work, so the total saving must exceed that extra cost.

Follow one round through the example

Start with the word “Please”. Suppose the large model, always choosing its highest-scoring next token, would continue “send the report”. Ordinary generation takes three sequential passes to produce those three tokens. For this illustration each word stands for one token; an actual tokeniser may split the text differently.

  1. The small model drafts “send the invoice today”. Those are proposals, not finished output.
  2. The large model scores the draft positions in one pass. In this example its preferred choices agree with “send” and “the”, so both are retained.
  3. At the next position, the large model prefers “report” to “invoice”. The system rejects “invoice” and uses “report” as the replacement. Its score is already available from that same large-model pass.
  4. The system also discards “today”. That guess depended on the rejected “invoice”, so it cannot simply be attached to the new text.
  5. The next round starts from “Please send the report”. The smaller model drafts again from that updated sequence.

This round produces three new tokens from one large-model pass plus the small model’s drafting work. Those counts describe the illustration. They are not a measured claim that the system is three times faster. If every draft token passes, the classic algorithm can also emit an additional target token before starting the next round.

What changes when answers are sampled

The example chooses the top-scoring token every time. Real generation often samples from a range of possible tokens instead. Speculative sampling uses the draft and target probabilities to decide whether to accept each guess. At the first rejection, it draws a replacement from an adjusted probability distribution.

The exact algorithm is designed to preserve the target model’s output distribution. In plain English, possible continuations retain the probabilities they would have under the target alone. That is different from promising the identical sentence on every run when random sampling is involved. Implementation details and numerical precision still matter.

When the extra model does not help

A draft model that frequently disagrees with the target creates work that is thrown away. A costly draft can consume the time saved by checking several tokens together. Keeping another model available also uses memory, and a busy serving system may have less spare calculation capacity for parallel verification.

The best draft length and model pairing depend on the workload. A long proposed block offers more potential progress but risks more wasted work. Some implementations use a small drafting component attached to the target instead of a separate complete model. The two-model example explains the core idea without representing every implementation.

What we would measure before adopting it

  • Time until the first visible token and time until the complete answer, measured separately.
  • Time spent drafting and verifying, alongside the number of accepted draft tokens per round.
  • Memory use and total serving cost at the expected number of simultaneous requests.
  • Results on representative tasks with the same target model and generation settings. A fast isolated demo does not establish the result under a busy workload.

We would compare the complete request with and without speculative decoding. Fewer target-model passes explain the opportunity; observed waiting time and cost determine whether it is worth using.

Sources and further reading