Speculative decoding in vLLM: when it speeds up generation

Speculative decoding makes vLLM faster only when the GPU has compute to spare: a few requests at a time, and text a small proposer can guess. On a server that already batches dozens of requests it adds work and can lower throughput. In vLLM 0.33.0, released on 2026-11-02, one flag turns it on: --speculative-config, a JSON object that names the method, eagle3, mtp, draft_model, ngram or suffix. With the standard verifier the output is still the main model's own, because every drafted token is checked before it is kept.

What speculative decoding is#

Speculative decoding is a draft-then-verify loop. A cheap proposer drafts the next few tokens, the main model scores all of them in one forward pass, and the server keeps the drafted tokens the main model agrees with plus one token of the main model's own. One pass can then emit several tokens instead of one.

It works because generating one token at a time leaves the GPU waiting on memory, not on arithmetic. vLLM's documentation describes the trade directly: at batch size 1 the GPU is memory-bound with spare compute, so the extra draft tokens are close to free.

The idea comes from two papers. Google's 2022 paper reported a 2 to 3 times speed-up on T5-XXL with identical outputs, and DeepMind's 2023 paper reported 2 to 2.5 times on a 70-billion-parameter model without compromising sample quality.

The methods in vLLM 0.33.0#

vLLM 0.33.0's documentation rates six ways to draft, and they differ in what does the drafting. The last two columns are its own rating of the gain:

eagle loads heads trained for the original EAGLE, and two more values, dflash and dspark, drive block drafters that ship with specific models such as DeepSeek-V4-Flash-DSpark.

Three things changed if you are following an older tutorial. The separate --speculative-model and --num-speculative-tokens flags are deprecated in favour of --speculative-config. Medusa and the MLP speculator were removed in 0.33.0. And model-specific MTP names such as deepseek_mtp are rewritten to mtp with a deprecation warning.

Turn it on with EAGLE-3#

The method I would try first is EAGLE-3, when a trained head exists for your model, because it adds almost nothing to the KV cache. Red Hat publishes one for Qwen3-8B under Apache-2.0, RedHatAI/Qwen3-8B-speculator.eagle3, so the example extends the Qwen3-8B server from deploying vLLM with Docker. Set up the VM, Docker and VLLM_API_KEY as that guide does.

Then start the 0.33.0 image with one more argument:

docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm \
  vllm/vllm-openai:v0.33.0 \
  Qwen/Qwen3-8B \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --api-key "$VLLM_API_KEY" \
  --speculative-config '{"method": "eagle3", "model": "RedHatAI/Qwen3-8B-speculator.eagle3", "num_speculative_tokens": 3}'

docker logs -f vllm

The JSON has three keys. method picks the proposer, model is the head or draft checkpoint, and num_speculative_tokens is how many tokens to draft per step. The value 3 is the default this head was published with. vLLM downloads the 2.04 GB head next to the main model on the first start. When the log shows Application startup complete, send requests exactly as before: clients need no change, and the model name stays Qwen/Qwen3-8B.

Check the driver before you pull. The 0.33.0 image is a CUDA 13.0 build that needs an R580 or newer driver, and vLLM no longer publishes CUDA 12.9 images: v0.32.0-cu129 is the last one. On an older driver the 0.33.0 image has a compatibility mode for R535 and R570 on select professional and datacenter GPUs, which you enable with -e VLLM_ENABLE_CUDA_COMPATIBILITY=1. Checking your driver and CUDA version takes one command.

One caveat belongs to this head, not to the method. Its model card says it was trained with Qwen3's thinking turned off, so measure the acceptance rate on your own prompts if you leave thinking on.

The other methods#

Every other method is the same command with a different JSON object. These configurations come from vLLM's own documentation:

A draft from a different model family needs "use_heterogeneous_vocab": true, which works with greedy draft sampling only. Gemma 4's assistant checkpoints go through mtp as well, with the assistant repo in model. The n-gram method sets both lookup bounds to 5 when you give neither.

Suffix decoding is the one that needs a custom image, because the official image does not include Arctic Inference and vLLM refuses to start without it. Two lines are enough:

FROM vllm/vllm-openai:v0.33.0
RUN uv pip install --system arctic-inference==0.1.1

I pin 0.1.1 because it is the version vLLM's error message names and its CUDA test requirements use. Arctic Inference 0.3.0 is newer, but vLLM's CUDA tests do not run against it. Version 0.1.1 is published only as a source archive, so the RUN line compiles it while the image builds.

For suffix decoding num_speculative_tokens is a ceiling, since the method drafts a different number of tokens at every step, and vLLM suggests 16 or 32. It also makes vLLM fall back to its older V1 model runner, which the startup log announces.

What each method costs in GPU memory#

The cost has two parts: the proposer's weights, and the KV cache its own layers keep for every token of context. Both come out of the same --gpu-memory-utilization budget, 0.92 of the card by default, so whatever the proposer takes is no longer there for the main model's context. For Qwen3-8B on a 48 GB card:

These are our calculations with the KV cache formula, 2 x layers x 8 KV heads x 128 x 2 bytes, for a card that reports 46,068 MiB with ECC on, before activations and CUDA graphs take their share. The EAGLE-3 row comes from reading vLLM's loading code, not from a run. vLLM drops a head's own token embedding and uses the target's only when the two are identical, and this head's copy, 1.24 GB of the 2.04 GB, differs from Qwen3-8B's (our check of the two files), so the whole file stays on the GPU. A head that ships without an embedding, or with an exact copy, would cost about 0.8 GB here.

The surprise is the draft model. Qwen3-0.6B is small in parameters, but it has 28 layers with the same KV width as the 36 layers of Qwen3-8B, so it adds 112 KiB per token. One 32,768-token request then needs 4.5 GiB for the main model and another 3.5 GiB for the draft. The EAGLE-3 head is larger on disk and adds only 4 KiB per token, because it is a single layer.

MTP heads ship inside the target's own checkpoint, so there is nothing extra to download, though their layers still need memory for weights and KV cache. The optional draft_lm_head_quantization key gives the drafter a quantized copy of the output layer, which vLLM's docs put at about 0.6 GiB in FP8 for a 248,000 x 2,560 head.

The line to read is the same as without speculation: GPU KV cache size: N tokens in the startup log, once with the config and once without. If the server no longer starts, fixing vLLM out-of-memory errors covers the options.

When it helps and when it slows the server down#

The rule I follow is to enable it for latency at low concurrency and to measure before trusting it under load. vLLM's documentation opens with the same condition: speculative decoding reduces inter-token latency under medium-to-low request rates on memory-bound workloads.

The reason is arithmetic. Every running request now sends its draft tokens through the main model on each step, so the effective batch is the number of requests times the draft length. vLLM's docs say that once that product passes a critical size, speculation makes each output token slower, and every rejected draft token is compute spent on nothing.

How much you gain depends on how many drafted tokens survive. Red Hat's card for the Qwen3-8B head reports mean acceptance lengths, at 3 draft tokens, of 2.39 on coding, 2.48 on math reasoning and 2.13 on summarization. Each verification step yields a bit over two tokens instead of one, and the step itself costs more than a plain one.

Published speed-ups, each from its own paper and its own setup:

Read the EAGLE-3 row twice. The same method has a headline of 6.5x and a figure of 1.38x at batch size 64, and that gap is why I do not quote one number for a server.

Measure it on your own traffic

Random-token prompts are the wrong test, because no proposer can guess the continuation of noise. vLLM's benchmark client reads Spec-Bench, real prompts in 13 categories from coding to summarization. Run it twice, against a server with the config and one without, at the concurrency you expect:

wget -q https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl
docker cp question.jsonl vllm:/tmp/question.jsonl

docker exec -e OPENAI_API_KEY="$VLLM_API_KEY" vllm vllm bench serve \
  --backend openai-chat --endpoint /v1/chat/completions \
  --model Qwen/Qwen3-8B \
  --dataset-name spec_bench --dataset-path /tmp/question.jsonl \
  --num-prompts -1 --max-concurrency 1

The server also reports acceptance itself. The log prints a line that starts SpecDecoding metrics: Mean acceptance length, and /metrics exposes vllm:spec_decode_num_draft_tokens_total and vllm:spec_decode_num_accepted_tokens_total. Starting the server with --per-request-spec-decode-metrics summary adds the same figures to each response.

Turn it down as load rises

A server whose load swings can keep speculation for quiet hours and switch it off when busy. The key is num_speculative_tokens_per_batch_size, a list of [first, last, tokens] entries by number of running requests, where 0 tokens means no drafting:

{
  "method": "eagle3",
  "model": "RedHatAI/Qwen3-8B-speculator.eagle3",
  "num_speculative_tokens": 3,
  "num_speculative_tokens_per_batch_size": [[1, 16, 3], [17, 64, 1], [65, 512, 0]]
}

The thresholds are placeholders for the ones your own benchmark finds. vLLM's docs list this as tested with EAGLE, EAGLE-3 and DFlash, their examples start the server with VLLM_USE_V2_MODEL_RUNNER=0, the older model runner, and it does not combine with data parallelism.

What stays the same, and the one lossy option#

The output distribution stays the main model's with the default settings. vLLM's docs call the standard rejection sampler lossless up to the precision of the hardware, and its tests check that greedy decoding with speculation matches greedy decoding without it. Small differences between runs can still appear, as they do without speculation, because vLLM does not promise stable log probabilities across batch sizes.

The exception is FLy, which you opt into with "rejection_sample_method": "fly". It accepts some tokens the standard check would reject, and vLLM's own page warns that it does not preserve the target distribution and may degrade outputs. The 0.33.0 release notes put its gain at 1.04 to 1.31 times decode throughput on the hardware they tested. I would leave it off unless you can evaluate answer quality on your own task.

Speculative decoding in llama.cpp#

llama.cpp has the same feature in llama-server, selected with --spec-type. In llama.cpp 0.9.0, released on 2026-11-04, a draft model is a second GGUF file passed with -md:

llama-server -m Qwen3-8B-Q8_0.gguf -md Qwen3-0.6B-Q8_0.gguf --spec-type draft-simple -ngl all

Both files are Qwen's own GGUF builds. Without a draft model, --spec-type ngram-mod drafts from repeated text in the conversation, and --spec-default turns that on with default settings. The other values cover EAGLE-3 (draft-eagle3), MTP heads (draft-mtp) and the block drafters. The server prints a draft acceptance rate line in its statistics, which is the number to watch. Running llama.cpp on a GPU covers the server itself.

Speculative decoding FAQ#

Does speculative decoding change the model's answers?

Not with the default verifier. The main model checks every drafted token, and vLLM tests that greedy output with speculation matches greedy output without it. Only the opt-in FLy verifier trades accuracy for speed.

How much extra GPU memory does it need?

It depends on the proposer. An EAGLE-3 head for Qwen3-8B is a 2.04 GB download that, by our reading of vLLM's loading code, stays on the GPU in full, plus 4 KiB of KV cache per token. A Qwen3-0.6B draft is 1.50 GB of weights and 112 KiB per token, which nearly doubles the KV cache. N-gram and suffix decoding load no model.

Does it work across several GPUs?

Yes, with tensor parallelism. Set draft_tensor_parallel_size in the JSON: it can be 1 or the target's value for a head, and it has to equal the target's for a standalone draft model. vLLM across several GPUs covers the target side.

What replaced --speculative-model?

--speculative-config, with the draft checkpoint in its model key. vLLM 0.33.0 also accepts --spec-method, --spec-model and --spec-tokens as shorthands for the three common keys.

My rule: turn on EAGLE-3 or MTP when a head exists for your model and the server handles a few requests at a time, use n-gram or suffix decoding for workloads that repeat text such as code edits and coding agents, and skip a standalone draft model unless you have the KV cache to spare. Then run the benchmark at your real concurrency before you keep it. If the numbers say no, a plain vLLM server on the same GPU is already the fast path.