vLLM vs llama.cpp: which server for which job

The rule I follow is: llama.cpp for one user or a small team, and for any model that has to spill into system RAM, and vLLM once requests overlap. The two servers are built for different jobs. vLLM is a GPU serving engine that batches many requests through one model and claims most of the card up front to do it. llama.cpp is an inference engine that runs one model on whatever hardware is there, GPU, CPU or both, from a single quantized file. Both speak the OpenAI API, so the client code is the same either way. What changes is the model file you download, how the GPU's memory is spent, and what happens when ten requests arrive at once.

vLLM and llama.cpp side by side#

The table compares vLLM 0.33.0, released on 2026-11-02, with llama.cpp v0.9.0, released on 2026-11-04, from their docs and source.

Continuous batching against slots#

Both batch, and the difference is scale. vLLM was designed around a paged KV cache and a scheduler that adds and removes requests between steps, so that hundreds of sequences share the GPU. By default its API server admits up to 256 requests at once on a 48 GB card or an A100, and 1,024 on cards with 70 GiB or more. Every one of them draws KV cache blocks from a single pool as it grows, so a short request costs little and nothing is reserved for tokens that never arrive.

llama-server has slots. Left at its defaults it starts 4 of them over one shared context, and continuous batching, which is on by default, decodes the active slots together. Start it with -c 32768 and 4 users share 32,768 tokens between them. You can raise -np, and you size -c to match, because the KV cache for the whole context is allocated when the server starts.

For one user the two designs converge: one sequence, one token at a time, and the speed is set by the GPU's memory bandwidth and the precision of the weights. The designs separate when requests overlap, which the benchmark section below puts a number on.

Safetensors against GGUF#

The file format follows from the server. vLLM loads Hugging Face checkpoints in safetensors and reads the quantization from each checkpoint's config, so a BF16, AWQ, GPTQ, FP8 or NVFP4 build of a model runs with no extra flag. llama.cpp loads GGUF, a single file that holds the weights, the tokenizer and the chat template, in quantization types from about 2 to 8 bits per weight. Neither server is a good host for the other's format: vLLM reads GGUF only through a plugin that its own docs call highly experimental and under-optimized.

For Qwen3-32B the choices look like this on Hugging Face:

The 8-bit files are the same size to within a few percent. The difference is the hardware each can use. FP8 computation needs compute capability 8.9 or newer, and on an Ampere card such as the RTX A6000 vLLM runs an FP8 checkpoint with 8-bit weights only. GGUF's Q8_0 and Q4_K_M run on any of them, and on a CPU. GGUF explained covers the quantization types, and vLLM quantization covers AWQ, GPTQ, FP8 and NVFP4.

How each one takes GPU memory#

vLLM claims a fraction up front. At startup it takes 92% of the GPU, loads the weights, and turns everything left into KV cache blocks. On a 48 GB card that reports 46,068 MiB, that is 41.4 GiB whether the model is 15 GiB or 32 GiB (our calculation). nvidia-smi shows the card nearly full with no request running, and a second process on the same GPU has to be planned for with a lower --gpu-memory-utilization. What you get for it is the pool: with Qwen3-8B in BF16, about 26.1 GiB of KV cache, room for about 190,000 tokens shared by every running request.

llama.cpp allocates for the context you ask for. It loads the weights, reserves a KV cache for exactly -c tokens, adds a compute buffer, and stops. Qwen3-32B at Q4_K_M with -c 32768 is 18.4 + 8.0 = 26.4 GiB, which leaves about 18.6 GiB of the same card free for a second model or another program. The other side of that is rigidity: the context is fixed at start, and a longer conversation than you planned for does not fit. llama.cpp on a GPU has the memory table for more models, and the KV cache guide has the formula both servers obey.

When the model is larger than the GPU#

This is llama.cpp's home ground. Its --fit logic is on by default: when a model and its context do not fit, it lowers a context you did not set, then keeps as many layers on the GPU as fit and runs the rest on the CPU from system RAM. For mixture-of-experts models, --cpu-moe and --n-cpu-moe keep the expert weights in RAM and the rest on the GPU. It is slow next to a full offload, but the model runs. I pass -ngl all when I do not want that, so a model that does not fit fails with an error instead.

vLLM refuses to start when the weights plus one full-length request do not fit, and prints why. It does have --cpu-offload-gb, which its source describes as a virtual way to increase GPU memory that needs a fast CPU-GPU interconnect, because part of the model is copied to the GPU on every forward pass. I treat that as an escape hatch, not a way to serve. If the model does not fit the card under vLLM, the answers are a smaller quantization, a bigger card or a second GPU, as fixing vLLM out-of-memory errors explains.

RAM matters for the llama.cpp route. A single RTX A6000 came with 24 to 64 GB of RAM in the 2026-09-27 catalog, so check the offer card before you plan to hold half a model there.

Several GPUs#

vLLM's multi-GPU story is the mature one. --tensor-parallel-size splits every layer across the GPUs of one node, and --pipeline-parallel-size splits the model along its layers. Its docs recommend pipeline parallelism on GPUs without NVLink, such as the L40S. vLLM across several GPUs covers the choice.

llama.cpp splits by layers by default (-sm layer): each GPU holds a contiguous run of layers and their KV cache, which is pipeline parallelism under another name and tolerates a slow link between cards. -ts 3,1 changes the proportions. It also has a tensor-parallel mode, -sm tensor, which its docs mark experimental: it needs flash attention and an unquantized KV cache, does not support every architecture, and switches --fit off. The older -sm row is deprecated. For fitting a large GGUF across two 48 GB cards the layer split is enough. For throughput across GPUs I would use vLLM.

Start-up and downloads#

On QuantaCloud every launch starts from an empty disk, so start-up begins with a download, and the two differ here too (our calculation from the registry and Hugging Face sizes). vLLM's image is 9.74 GB and llama.cpp's CUDA server image is 2.60 GB. With Qwen3-32B, a fresh instance pulls 44.06 GB for vLLM with the FP8 checkpoint, against 37.42 GB for llama.cpp at Q8_0 or 22.36 GB at Q4_K_M.

After the download, vLLM loads the weights, measures its memory and builds the KV pool before it answers, and llama.cpp memory-maps the GGUF file and copies it to the GPU. Neither project publishes start-up times, and I have not measured them, so I will not put a number on it.

Structured output and tool calling#

Both constrain output to a JSON schema through the OpenAI field response_format, so a client that sticks to that field works on either. Beyond it they diverge. vLLM takes a structured_outputs object with a JSON schema, a regular expression, a list of choices or a grammar, enforced by xgrammar or guidance. llama-server takes a json_schema or a grammar in llama.cpp's own GBNF notation.

Tool calling is where the defaults differ most. In llama-server it works out of the box: the Jinja chat template is on by default, llama.cpp recognises the native tool formats of common model families, and it falls back to a generic format for the rest. In vLLM you turn it on at start with --enable-auto-tool-choice and name a --tool-call-parser, which has to match the tool-call format of the model family.

What a published benchmark shows#

The clearest published comparison is from Red Hat's developer blog, dated 2025-09-30. On one NVIDIA H200 it served Llama 3.1 8B at 16 bits on both engines, vLLM 0.10.0 and llama.cpp b6100, and ramped a load generator from 1 to 64 simultaneous users. At peak load vLLM delivered more than 35 times the requests per second and more than 44 times the output tokens per second.

Read that with two caveats. The article does not say how many slots llama-server ran with, and in that build the default was 1, so the gap may partly measure a queue. And the builds are more than a year old: today's llama-server starts with 4 slots. The direction is not in doubt, since it follows from the designs. The size of the gap depends on how both servers are configured.

Running each on QuantaCloud#

Both run on the Bare Metal template, a plain Ubuntu 22.04 VM with the NVIDIA driver and Docker. QuantaCloud has no vLLM or llama.cpp template and no managed endpoint, so you start the server yourself. A 48 GB RTX A6000 is enough to try either.

For vLLM, deploying vLLM with Docker has the full setup with the driver check and an API key, and installing vLLM covers pip and uv. For llama.cpp, running llama.cpp on a GPU does the same with the official CUDA image. Neither should face the internet as installed. vLLM listens on every interface unless you pass --host, and its API key covers only the /v1-style routes. llama-server listens on 127.0.0.1 by default, but its Docker image binds every interface inside the container, so publish the port as 127.0.0.1:8080:8080. Stopping an instance deletes its disk, so each launch downloads the image and the weights again.

vLLM vs llama.cpp FAQ#

Is vLLM faster than llama.cpp?

Under concurrent load, yes, and by a wide margin in Red Hat's published test: it measured more than 44 times the output tokens per second at peak load on an H200, with the caveats above. For a single request the engines are much closer, and the precision of the weights matters more than the server.

Can vLLM run GGUF files?

Only through vllm-gguf-plugin, which vLLM 0.33.0's docs describe as highly experimental and under-optimized. Use the model's safetensors checkpoint with vLLM, and keep GGUF for llama.cpp or Ollama.

Can llama.cpp serve several users at once?

Yes. llama-server starts with 4 slots and batches them together, and -np raises the count. Size -c for all of them, because the slots share one context by default. For a team of a few people that is enough. When requests overlap all day, move to vLLM.

Which one needs less GPU memory?

llama.cpp, for the same model. It runs 4-bit and 5-bit GGUF files that vLLM has no direct equivalent for, it allocates only the context you ask for, and it can leave layers in system RAM. vLLM takes 92% of the card by design and uses it to hold more requests.

Where does Ollama fit?

Ollama runs llama.cpp's llama-server underneath and adds a model library and its own API. llama.cpp vs Ollama and vLLM vs Ollama cover those two comparisons.

My rule: start with llama.cpp when the users are a few people or one script, when the model only exists as GGUF, or when it has to spill into RAM. Move to vLLM the day requests start to overlap, and give it a checkpoint in BF16, AWQ or FP8 that leaves room for the KV cache. Either one starts on an RTX A6000 at $0.48/GPU-hr, and self-hosted LLM inference puts both next to the other serving options.