llama.cpp vs Ollama on a GPU server

The short answer is: on a GPU server they are the same engine. Ollama 0.40.16 starts llama.cpp's own llama-server for every GGUF model it runs, compiled from a llama.cpp build that Ollama pins. So the choice is not between two inference engines. It is between running llama-server yourself, with every flag in your hands, and letting Ollama manage the files, pick the flags and load models on demand. I run llama.cpp directly when one model serves an application, and Ollama when people want to try models without thinking about files.

How Ollama and llama.cpp relate today#

Ollama began as an application built on llama.cpp, spent 2025 building an engine of its own, and then came back. Its release notes tell the story. From 0.7.1 in May 2025 to 0.17.1 in February 2026 they announce one model architecture after another running on "Ollama's engine". In March 2026, version 0.19.0 put Apple silicon on MLX as a preview. Then 0.30.0, released on 2026-05-13, announced "improved compatibility and performance using llama.cpp" for other hardware, and since 0.40.0 the models MLX supports run on it by default on Apple silicon.

The 0.40.16 source shows where that ended up. A file named LLAMA_CPP_VERSION pins the llama.cpp build, b12177. Ollama compiles that build's llama-server with its own compatibility patches, starts one llama-server process on a local port for each loaded GGUF model, and talks to it over HTTP. The directories that held Ollama's Go engine are no longer in the repository. What Ollama adds is everything around the engine: a model library, a scheduler that loads and unloads models, its own API, and an MLX runner for safetensors models.

On a Linux server with an NVIDIA GPU, a GGUF model in Ollama therefore runs on llama.cpp. ollama ps has a RUNNER column if you want to see it for a loaded model.

llama.cpp and Ollama side by side#

The defaults below are from llama.cpp v0.9.0, released on 2026-11-04, and Ollama 0.40.16, released on 2026-11-09.

Model files and where they come from#

Both run GGUF files, and the difference is who fetches them. llama-server takes a file path with -m, or downloads from Hugging Face with -hf Qwen/Qwen3-32B-GGUF:Q4_K_M, which picks Q4_K_M when you name no quantization. You choose the repository, so you choose who converted the file.

Ollama pulls from its own library by name. ollama pull qwen3:32b fetches a 20.2 GB GGUF layer that the registry labels Q4_K_M, plus small layers for the prompt template, the licence and the parameters. Outside the library there are two routes. ollama run hf.co/Qwen/Qwen3-32B-GGUF:Q4_K_M runs a GGUF repository from Hugging Face directly, and a Modelfile with FROM /path/to/file.gguf imports a file you already have. Ollama does not quantize a GGUF on import, and since 0.34.1 it leaves converting safetensors to GGUF to llama.cpp's own tools. GGUF explained covers the format, the quantization types and the conversion.

Default context, and a model that does not fit#

Context length is where the two behave most differently, because Ollama decides it for you. Ollama picks a default from the total memory your GPUs report: 4,096 tokens below 23 GiB, 32,768 from 23 GiB and 262,144 from 47 GiB, capped at the model's trained context. It then hands that number to llama-server as an explicit -c.

llama-server on its own starts from the model's trained context. With --fit, which is on by default, it lowers that context towards 4,096 tokens if memory is short, and only then moves layers to the CPU, keeping 1,024 MiB of each GPU free. It adjusts only what you left unset. That has a consequence inside Ollama: the context arrives already set, so the one thing left to adjust is how many layers stay on the GPU.

A 70B model shows the effect (our calculation). Llama 3.3 70B at Q4_K_M is a 39.6 GiB file with a trained context of 131,072 tokens, and its KV cache takes 320 KiB per token. A 48 GB card that reports 49,140 MiB crosses Ollama's 47 GiB line, so the default context is the model's full 131,072 tokens: 40 GiB of cache on top of 39.6 GiB of weights. A card that reports 46,068 MiB gets 32,768 tokens, which is still 49.6 GiB in total. Either way the model no longer fits the GPU. Layers go to system RAM, and the PROCESSOR column of ollama ps shows a split such as 48%/52% CPU/GPU. If the load fails for lack of memory, Ollama retries once with the next lower default.

The fix is the same in both: a context that fits. OLLAMA_CONTEXT_LENGTH=8192 for Ollama, or -c 8192 -ngl all for llama-server, comes to 39.6 + 2.5 = 42.1 GiB. With llama.cpp I always pass both flags, because -ngl all makes a model that does not fit fail with an error instead of quietly running on the CPU. Ollama context length goes through Ollama's side in detail, and Ollama not using the GPU covers the CPU split.

Several models and concurrent requests#

Ollama is built to hold several models and serve each one request at a time. It loads a model when a request names it, keeps up to 3 per GPU if they fit, and unloads one 5 minutes after its last request. Each model works on 1 request at a time unless you raise OLLAMA_NUM_PARALLEL, and up to 512 more wait in a queue before Ollama answers with a 503. Parallel requests cost memory, because Ollama multiplies the context by the parallel count: 4 parallel requests at 32,768 tokens start llama-server with -c 131072 -np 4.

llama-server is one model per process by default, with 4 slots that share a single context. Started with -c 32768, its 4 slots draw on the same 32,768 tokens, and continuous batching runs them together. For several models it has a router mode: start it without a model and it loads the model each request names, up to 4 at once. llama-swap does the same job with more control over which models may stay loaded together.

Control over GPU offload#

This is the real cost of Ollama's convenience. llama-server exposes the placement of a model flag by flag: -ngl for the number of layers on the GPU, -dev to pick devices, -sm and -ts to split across several GPUs, --cpu-moe and --n-cpu-moe to keep mixture-of-experts weights in system RAM, and -ctk and -ctv to set the KV cache type for that one model.

Ollama exposes two of those per model, as the options num_gpu and main_gpu. The KV cache type is one environment variable for the whole server (OLLAMA_KV_CACHE_TYPE), and with several GPUs Ollama decides the placement: one GPU if the model fits, all of them if it does not. For one 48 GB card that is rarely a problem. For a model you have to shape by hand, such as a mixture-of-experts model larger than the GPU, llama.cpp directly is the tool.

The API each one exposes#

Both speak the OpenAI API, so most clients need only a base URL. llama-server serves /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings and /v1/models, an Anthropic-style /v1/messages, and its native /completion and /tokenize endpoints. It checks an API key if you start it with --api-key.

Ollama's first API is its own, under /api: generate, chat, pull, list, show and ps. Its /v1 is a subset of the OpenAI API, without logprobs, tool_choice or n, and it cannot set the context length, which you bake into a Modelfile instead. It also serves a subset of Anthropic's /v1/messages. The local Ollama API has no authentication at all, so it must stay on 127.0.0.1. The Ollama API on a remote GPU shows how to reach it safely.

How fast each picks up new models and llama.cpp features#

The gap is days, not months. llama.cpp publishes several nightly builds a day, 14 of them on 2026-11-11, and an official release every week or two: v0.8.0 on 2026-10-28 and v0.9.0 on 2026-11-04. Ollama 0.40.16 shipped on 2026-11-09 with build b12177, which was three days old. By 04:10 UTC on 2026-11-12, nightly build b12307 was already 130 commits ahead of it. Ollama's release notes record five llama.cpp updates between 2026-10-12 and 2026-11-09.

So freshness is not what you give up with Ollama. What you wait for is narrower. A new model has to arrive in Ollama's library, or as a GGUF on Hugging Face that the pinned build can read. And a new llama-server flag does nothing for you until Ollama passes it. With llama.cpp you pull the build that has the fix on the day it lands.

Running each on QuantaCloud#

The Open WebUI + Ollama template is the quickest Ollama path: Ollama runs in the app container beside Open WebUI, behind your QuantaCloud login and Open WebUI's own sign-in. Open WebUI on QuantaCloud describes it, and Ollama in Docker covers a plain Ollama server on the Bare Metal template.

Run ollama --version inside the template before you rely on anything version-specific here: what this page says about llama-server applies from Ollama 0.30.0.

llama.cpp runs on the Bare Metal template, a plain Ubuntu 22.04 VM with the NVIDIA driver and Docker, from its official CUDA image.

This starts the model Ollama calls qwen3:32b, with the context and the offload set by hand:

export LLAMA_API_KEY=$(openssl rand -hex 32)

docker run -d --name llama --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e LLAMA_API_KEY \
  ghcr.io/ggml-org/llama.cpp:server-cuda-b12110 \
  -hf Qwen/Qwen3-32B-GGUF:Q4_K_M --alias qwen3-32b \
  -ngl all -c 32768

server-cuda-b12110 is the image of llama.cpp v0.9.0, a 2.6 GB download. Inside the container the server listens on port 8080, and the 127.0.0.1: in the port mapping keeps it off the public interface. A llama-server binary you build yourself listens on port 9931 since October 2026, so pass --port if your scripts expect 8080. Running llama.cpp on a GPU walks through the rest: checking the offload line in the log, the API calls and a source build.

Stopping a QuantaCloud instance deletes its disk, so either server downloads its models again on the next launch: 19.76 GB for this file.

llama.cpp vs Ollama FAQ#

Does Ollama use llama.cpp?

Yes. Ollama 0.40.16 pins llama.cpp build b12177 and starts its llama-server for every GGUF model. Its other runner, MLX, is for safetensors models and is the default on Apple silicon.

Is llama.cpp faster than Ollama?

For the same GGUF file on the same GPU, the code that generates the tokens is the same, a few days apart in age. Differences come from what surrounds it: the context Ollama chooses, whether every layer is on the GPU, and the build each one runs. Check ollama ps for a CPU share before you blame the engine.

Can Ollama run a GGUF file I already downloaded?

Yes. Write a Modelfile with FROM /path/to/file.gguf and run ollama create. For a GGUF repository on Hugging Face, ollama run hf.co/<user>/<repo> skips the Modelfile.

Can I use Open WebUI with llama.cpp instead of Ollama?

Yes. Open WebUI connects to llama-server as an OpenAI connection: add the server's URL ending in /v1 under Settings, Admin, Connections, OpenAI. Open WebUI with Ollama covers the Ollama pairing.

Can either one serve many users at once?

Within limits. llama-server runs its slots together with continuous batching, and Ollama raises the slot count with OLLAMA_NUM_PARALLEL, at the price of a full context for every slot. When requests overlap all day, vLLM is the usual answer, and vLLM vs llama.cpp and vLLM vs Ollama explain why.

My rule: Ollama when the job is trying models, llama.cpp when the job is running one. Ollama gives you a library and sensible defaults on top of the same engine, and it costs you the flags. The day you need to set the context, the offload or the build yourself, start llama-server with -ngl all and a fixed -c on an RTX A6000 at $0.48/GPU-hr.