Speculative Decoding Is Why Local Models Feel Instant

Local LLMs used to feel like a slide show: one token, pause, one token. The models did not get that much faster. The serving stack did. Speculative decoding runs a cheap draft model that proposes a handful of tokens, then asks the larger model to verify them in a single forward pass. If the draft was right, you just got several tokens for the price of one. If it was wrong, you keep the prefix that matched and continue.

This is why an 8B or 14B model on a 16 GB card now feels like a cloud chat, especially on code and other predictable text. Acceptance rates are high when the draft and the target were trained on similar data. They drop on creative writing and on languages the draft barely knows, which is why some stacks swap draft models per task instead of using one tiny network for everything.

The practical takeaway for anyone running llama.cpp or a local router: tokens per second is no longer just 'how big is the model.' It is how often the draft is right. A slightly worse small model that agrees with the large one will beat a 'smarter' draft that keeps getting rejected.