Speculative Decoding Clearly Explained

(done right, it can make LLM inference 4x faster)

A local LLM typically produces one token per model pass. Each pass reads the model’s weights from memory, often leaving compute units waiting for data.

Generating a sentence means e process for every token.

Now introduce a smaller model that can predict the next token quickly. Let it make several predictions ahead while the larger model waits to check them.

The smaller model is the draft model. The larger model, called the target model, is the LLM whose output we want to preserve.

The draft model proposes. The target model verifies.

Because the guesses are already available, the target can check several positions in one pass instead of generating each token separately.

Here’s how that works.

→ Draft a possible continuation.

After “The cat sat,” the draft model proposes “on the red mat.” These tokens are provisional.

→ Check the guesses together.

The target model scores all four positions in one forward pass. Each position uses the preceding draft tokens as context.

→ Accept from left to right.

For sampling, let p be the target model’s probability for a guess and q the draft model’s probability. Accept it with probability min(1, p/q), stopping at the first rejection.

→ Correct and continue.

Discard the rejected token and all later guesses. Sample a replacement from the positive difference between the target and draft distributions, normalized into probabilities.

In the diagram, “on” and “the” are accepted. “Red” is rejected, and the replacement is “big.”

That produces three new tokens from one target-model pass. If every guess is accepted, the target can supply an additional token.

This acceptance and correction procedure preserves the target model’s output distribution. The speedup comes from replacing expensive passes with cheaper drafting and shared verification.

Uzu, an open-source engine for Apple Silicon, takes this further. DFlash proposes candidates in parallel, Weaver builds coherent draft paths, and rollback-free verification checks branches without repeatedly restoring the model’s running state.

In our base M5 coding test, 4-bit Qwen3.5 9B reached 92 to 117 tokens per second with Uzu, compared with 22 using plain llama.cpp. These complete engine results also reflect quantization and hardware-specific kernels.

Explore Uzu on GitHub: github.com/trymirai/uzu

(don't forget to star 🌟)

My quoted article explains how Uzu combines these techniques and how to benchmark the result on your own prompts.