Autoregressive language models usually generate one token at a time. Each new token depends on the previous result, which makes decoding a frequent bottleneck in local inference.
Draft and verify
Speculative decoding changes the sequence. A lightweight draft model proposes several tokens. The larger target model evaluates that proposal in a single pass and accepts the prefix that matches its own distribution. When the proposal is good, the target model advances by multiple tokens instead of one.
Why it matters
The method can improve perceived responsiveness without changing the target model’s accepted output. It is especially useful when the target runtime is expensive and a small draft model is cheap enough to run alongside it.
Limits
The gain depends on how often the draft model proposes useful tokens, the cost of verification and the runtime’s batching behaviour. A poor draft model, short output or a memory-constrained device can reduce or remove the benefit. As with every runtime technique, end-to-end measurement matters.
Outlook
Speculative decoding is a good example of architectural efficiency: a system can become faster through cooperation between models, not only through larger hardware or lower precision.