Install llama-cpp-python with CUDA on a GPU server

The reason llama-cpp-python ignores your GPU is almost always the install. pip install llama-cpp-python downloads the source and compiles llama.cpp for the CPU, because CUDA is off unless you ask for it, and the bindings then put no layers on the GPU unless you ask for that too. The fix has two parts: install a CUDA build, either from source with CMAKE_ARGS="-DGGML_CUDA=on" or as one of the project's prebuilt CUDA wheels, and load the model with n_gpu_layers=-1. Then one line in the load log proves it worked. The commands below are for llama-cpp-python 0.3.36, released on 2026-10-01, on an Ubuntu 22.04 GPU VM.

Why pip gives you a CPU-only build#

Four things go wrong, and they stack:

The third one surprises people who come from llama.cpp itself, where llama-server tries to put every layer on the GPU by default. The bindings do the opposite: 0 means none, and -1 means all of them.

Launch a GPU VM#

Use the Bare Metal template: Ubuntu 22.04 with the NVIDIA driver and Docker. A 48 GB RTX A6000 is plenty for the 8B model used to check the install.

Most single-GPU VMs are running in about 3 minutes (median). Connect as ubuntu (connecting over SSH) and read three things: the GPU, its compute capability, and the CUDA version the driver supports, which nvidia-smi prints in the top right of its table.

ssh ubuntu@YOUR_INSTANCE_IP
nvidia-smi --query-gpu=name,driver_version,compute_cap --format=csv
nvidia-smi | head -4

Both routes need NVIDIA's apt repository, so add it once:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update

The two routes compare like this:

The CPU row matters only if part of a model runs on the CPU. The project's README recommends the source build for that reason: llama.cpp is compiled with optimizations specific to your system, and prebuilt binaries have to switch them off.

Route 1: build from source with GGML_CUDA#

GGML_CUDA is the CMake option that turns the CUDA backend on, and CMAKE_ARGS is how llama-cpp-python passes CMake options into the build. The build needs NVIDIA's CUDA toolkit, which includes the nvcc compiler, plus a C and C++ compiler and CMake. CUDA 12.8 suits an R570 or newer driver:

sudo apt-get install -y cuda-toolkit-12-8 build-essential cmake python3-venv
export PATH=/usr/local/cuda-12.8/bin:$PATH

python3 -m venv ~/llama-env
source ~/llama-env/bin/activate
pip install --upgrade pip

CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=86" CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) \
  pip install "llama-cpp-python[server]==0.3.36" --no-cache-dir --force-reinstall --verbose

Each flag earns its place. -DCMAKE_CUDA_ARCHITECTURES=86 compiles kernels for the RTX A6000 only. Use 80 for the A100, 89 for the L40, L40S and RTX 6000 Ada, 90 for the H100 and H200, and 120 for the RTX PRO 6000. Code built for 8.6 also runs on the 8.9 cards, but not on an A100, so build for the GPU you will run on. Without the flag, CMake older than 3.24 compiles llama.cpp's whole default list of architectures, which takes longer, and Ubuntu 22.04 ships 3.22.1. CMAKE_BUILD_PARALLEL_LEVEL lets CMake use every vCPU, and a single RTX A6000 came with 6 to 12 of them in the 2026-09-27 catalog. --no-cache-dir stops pip from reusing a CPU build it made earlier, --force-reinstall replaces one that is already installed, and --verbose shows the CMake log. Look for these three lines in it:

-- CUDA Toolkit found
-- Using CMAKE_CUDA_ARCHITECTURES=86 CMAKE_CUDA_ARCHITECTURES_NATIVE=...
-- Including CUDA backend

Older guides use -DLLAMA_CUBLAS=on. In the llama.cpp that 0.3.36 bundles, that option stops the configure step with an error that points to GGML_CUDA, and -DLLAMA_CUDA=on still works but warns.

Route 2: install a prebuilt CUDA wheel#

The project publishes CUDA builds of each release in its own package index, one per CUDA version, at https://abetlen.github.io/llama-cpp-python/whl/<cuda-version>:

The README lists Python 3.10, 3.11 and 3.12 for these wheels, so Ubuntu 22.04's Python 3.10 is covered. Pick the newest index whose CUDA version is not above the one nvidia-smi printed: cu125 on any CUDA 12 driver, or a cu13 index on R580 or newer, which is less than half the download. Checking your GPU, driver and CUDA version explains what that number means.

The wheels bundle llama.cpp but leave out the CUDA runtime and cuBLAS, so install those first, at the wheel's CUDA version. For cu125:

sudo apt-get install -y cuda-cudart-12-5 libcublas-12-5 python3-venv
sudo ldconfig
ldconfig -p | grep -E "libcudart.so.12|libcublas.so.12"

python3 -m venv ~/llama-env
source ~/llama-env/bin/activate
pip install --upgrade pip
pip install "llama-cpp-python[server]==0.3.36" --only-binary llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu125

--only-binary llama-cpp-python is the flag that keeps this route honest. pip searches PyPI and the extra index with no priority between them, and PyPI has the source of every version. Without the flag, a version or platform the index has no wheel for makes pip fall back to that source and compile a CPU build without an error. With it, the install fails instead. Pinning ==0.3.36 keeps the install repeatable. The cu125 wheel unpacks to 6.54 GB of files and a CUDA 13 wheel to 0.98 GB, so budget the disk.

Prove the offload#

The first check needs no model. llama_supports_gpu_offload() returns True only when the library has a working GPU backend and can see a GPU:

python -c "import llama_cpp; print(llama_cpp.__version__, llama_cpp.llama_supports_gpu_offload())"

It should print 0.3.36 True. False means a CPU-only build, or a CUDA build that cannot see the GPU. An import error that names libcudart.so.12 or libcublas.so.12 means the CUDA libraries are missing, as in route 2.

Then load a model with every layer on the GPU and read the log. Qwen3-8B-Q4_K_M.gguf from Qwen's own repository is 5.03 GB:

mkdir -p ~/models && cd ~/models
curl -L -O https://huggingface.co/Qwen/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4_K_M.gguf

from llama_cpp import Llama

llm = Llama(
    model_path="/home/ubuntu/models/Qwen3-8B-Q4_K_M.gguf",
    n_gpu_layers=-1,
    n_ctx=8192,
)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain a GGUF file in two sentences."}],
    max_tokens=1000,
)
print(out["choices"][0]["message"]["content"])

verbose is on by default, so llama.cpp's load log goes to stderr. These lines are the proof:

load_tensors: offloaded 37/37 layers to GPU
load_tensors:        CUDA0 model buffer size = ...
llama_kv_cache:      CUDA0 KV buffer size = ...

Qwen3-8B has 36 layers, and the 37th is the output layer. If the first number is lower, part of the model is on the CPU. If there is no offloaded line and only a CPU buffer, the build has no GPU backend. nvidia-smi in a second terminal should show a little more than 5.8 GiB in use: 4.68 GiB of weights and 1.13 GiB of KV cache for 8,192 tokens (our calculation), plus llama.cpp's compute buffer. Qwen3 writes its thinking before the answer, and the bindings return both as one text, so leave room in max_tokens.

Two more defaults differ from llama.cpp's own server. n_ctx is 512 unless you set it, and 0 means the model's trained context, 40,960 tokens for Qwen3-8B. And Flash Attention is off unless you pass flash_attn=True, which a quantized V cache (type_v) needs.

Run the bundled OpenAI-compatible server#

The [server] extra installed above adds a FastAPI server with the OpenAI routes. Its flags are the setting names with underscores, and it needs --n_gpu_layers -1 as well:

export LLAMA_API_KEY=$(openssl rand -hex 32)
echo "$LLAMA_API_KEY"

python -m llama_cpp.server --model ~/models/Qwen3-8B-Q4_K_M.gguf --model_alias qwen3-8b \
  --n_gpu_layers -1 --n_ctx 8192 --api_key "$LLAMA_API_KEY"

It listens on localhost, port 8000, and the key goes in as a Bearer token. From a second SSH session, with the same key exported:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $LLAMA_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen3-8b", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 1000}'

Keep it on localhost and reach it from your laptop with ssh -N -L 8000:127.0.0.1:8000 ubuntu@YOUR_INSTANCE_IP, as the SSH and VS Code guide describes. The README's --host 0.0.0.0 would put it on the VM's public address.

This server is built for one client. It runs one request at a time behind a lock. With its default interrupt_requests setting, a request that arrives while another reply is streaming ends that stream early. That is fine for a script or a notebook, and wrong for anything two people share.

How far the bindings trail llama.cpp#

llama-cpp-python bundles a fixed llama.cpp commit and moves it forward with each release, so it is always some weeks behind. On 2026-11-12 the gap looked like this:

The bindings' last three releases came 36 and 45 days apart, and 42 days had passed since the last one. The cost is model support and fixes. DeepSeek V4.1 Flash support was merged into llama.cpp on 2026-10-25, and the FP8 tensor type arrived with v0.9.0, so a GGUF that needs either does not load in 0.3.36. Before you download a new model for the bindings, check that its llama.cpp support predates the commit the bindings' changelog names.

When plain llama-server is the better choice#

My rule is simple: use the bindings when the model runs inside your own Python program, and llama-server when anything else talks to it. In a script, a notebook or an evaluation loop, the bindings keep everything in one process with no HTTP hop. For an endpoint, llama-server is the better tool on every count that matters on a rented GPU:

Running llama.cpp on a GPU sets up llama-server with the official CUDA image, and both servers speak the same OpenAI API, so a client moves between them with a new base URL. vLLM vs llama.cpp covers the point where neither is enough.

Before you stop the instance#

Stopping a QuantaCloud instance deletes its disk: the CUDA libraries, the virtual environment, the compiled build and the model. The next launch starts from scratch, which is the strongest argument for the prebuilt wheel if you launch often. Keep the install commands in a script on your laptop. The first hour is charged at launch, and the unused seconds are refunded when you stop.

llama-cpp-python FAQ#

Why is llama-cpp-python not using my GPU?

Check in this order. llama_cpp.llama_supports_gpu_offload() returning False means a CPU-only build: reinstall with CMAKE_ARGS="-DGGML_CUDA=on" and --no-cache-dir --force-reinstall, or install a CUDA wheel. If it returns True, you probably left n_gpu_layers at its default of 0. If the offloaded line shows fewer layers than the model has, the model and context do not fit the GPU.

Which CMake flag enables CUDA in llama-cpp-python?

-DGGML_CUDA=on, passed through CMAKE_ARGS or pip install -C cmake.args="-DGGML_CUDA=on". The old -DLLAMA_CUBLAS=on now stops the build with an error.

Do I need the CUDA toolkit for the prebuilt wheels?

Not the compiler, but the runtime and cuBLAS libraries, which the wheels expect to find on the system: libcudart.so.12 and libcublas.so.12 for the CUDA 12 wheels, from cuda-cudart-12-5 and libcublas-12-5 for cu125. The source build needs the full toolkit.

Which Python versions does llama-cpp-python support?

The package itself needs Python 3.8 or newer. The README lists 3.10, 3.11 and 3.12 for the prebuilt CUDA wheels. Ubuntu 22.04 ships 3.10, which is on that list.

Can llama-cpp-python load any GGUF file?

Any file the llama.cpp version inside it can read, which is build b11312 from 2026-10-01 for 0.3.36. GGUF explained covers the format and which quantization to download.

My rule for llama-cpp-python on a rented GPU: install a CUDA build on purpose, with GGML_CUDA or a pinned CUDA wheel and --only-binary, load every model with n_gpu_layers=-1, and read the offloaded line before anything else. When the model has to serve anyone but your own script, put it behind llama-server instead. An RTX A6000 at $0.48/GPU-hr runs either.