Ollama falls back to the CPU without an error, so finding out why starts with two readings. The PROCESSOR column of ollama ps says where the loaded model is running: on the GPU, on the CPU, or split between them. The inference compute line in the server log says whether Ollama found a GPU at all. A split with a GPU present means the model and its context do not fit in GPU memory, and on a GPU server that is the cause to check first. No GPU in the log means the driver, the container or an environment variable is hiding it. This page follows Ollama 0.40.16, released on 2026-11-09, and quotes its log messages from the source.
Read ollama ps first#
ollama ps lists the models that are in memory, so send the model a request and then run it on the server. In Docker the command runs inside the container:
ollama ps
sudo docker exec ollama ollama ps
In 0.40.16 the columns are NAME, ID, SIZE, PROCESSOR, CONTEXT, RUNNER and UNTIL. PROCESSOR is the one that answers the question:
| PROCESSOR | What it means | Where to go next |
|---|---|---|
100% GPU | The whole model is in GPU memory | Placement is fine. Nothing on this page applies |
48%/52% CPU/GPU | Part of the model sits in system memory and runs on the CPU. The first figure is the CPU share and the second the GPU share | The first cause below: it does not fit |
100% CPU | Nothing was loaded onto a GPU | The server log, to see whether Ollama found a GPU |
The other columns help with the diagnosis. SIZE is the memory the model takes together with its cache, CONTEXT is the context it was loaded with, and RUNNER names the engine, llamacpp for a GGUF model on Linux. An empty list means nothing is loaded: Ollama unloads a model 5 minutes after its last request unless you changed the keep-alive. On the Open WebUI + Ollama template, Ollama runs inside the app container, and the setup guide shows how to run the command there.
Then read the server log#
The server log is where Ollama says what it found and what it did with each model. Where it is depends on how Ollama runs:
| How Ollama runs | Where the log is |
|---|---|
| As the systemd service the install script creates | journalctl -u ollama --no-pager |
In a Docker container named ollama | sudo docker logs ollama |
As ollama serve in a terminal | In that terminal |
One search pulls out the lines that matter:
journalctl -u ollama --no-pager | grep -E "inference compute|default context|offloaded|driver too old|visible devices|compiled architectures|reducing automatic context"
sudo docker logs ollama 2>&1 | grep -E "inference compute|default context|offloaded|driver too old|visible devices|compiled architectures|reducing automatic context"
| Message in the log | When | What it tells you |
|---|---|---|
inference compute | At startup, once per GPU | With a GPU: library=CUDA, the GPU's name, its compute capability and its total and available memory. Without one: id=cpu library=cpu |
vram-based default context | At startup | The GPU memory Ollama counted, as total_vram, and the context it will use by default |
server config | At startup | Every environment variable as the server read it, CUDA_VISIBLE_DEVICES included |
gpu memory | Each time a model loads | The free and available memory of each GPU just before the load |
starting llama-server | Each time a model loads | The full command Ollama started, with the context after -c and, if a layer count was set, -ngl |
offloaded N/M layers to GPU | Each time a model loads | How many layers llama.cpp put on the GPU. Two equal numbers mean all of them |
The startup line is the first thing to look at. With a GPU it has this shape, and without one the second:
msg="inference compute" id=<id> filter_id=<id> library=CUDA compute=<capability> name=<device> description="<GPU model>" libdirs=<builds> driver=<version> pci_id=<id> type=discrete total="<size>" available="<size>"
msg="inference compute" id=cpu library=cpu compute="" name=cpu description=cpu libdirs=ollama driver="" pci_id="" type="" total="<size>" available="<size>"
For more detail, set OLLAMA_DEBUG=1 on the server and restart it. Ollama environment variables shows where to set it under systemd and in Docker.
The causes, in the order they happen#
Those two readings point to one of six causes. On a GPU server I check them in this order:
| What you see | Cause | Fix |
|---|---|---|
A CPU/GPU split in ollama ps, and a GPU in the log | The model plus its context does not fit in GPU memory | A shorter context, a q8_0 cache, a smaller quantization or a larger GPU |
nvidia-smi fails on the host, and inference compute says library=cpu after a restart | The driver changed under the running system | Reload the driver or reboot, and hold the driver packages |
sudo docker exec ollama nvidia-smi fails | The container was created without the GPU | Create it again with --gpus=all |
user overrode visible devices in the log | An environment variable hides the GPU | Unset it and restart the server |
NVIDIA driver too old, or compute capability not in compiled architectures | The GPU or its driver is too old for the bundled CUDA build | A newer driver. Below compute capability 5.0 there is no fix |
100% CPU for one model only, and -ngl 0 in its starting llama-server line | A num_gpu parameter limits the layers | Remove it from the Modelfile or the client |
The model and its context do not fit
This is the cause where nothing is broken. Ollama starts llama.cpp with the context fixed and the layer count left open, and llama.cpp puts as many layers on the GPU as fit while keeping 1,024 MiB free. For a model with a vision projector, such as qwen3.8, Ollama raises that margin to the projector's size plus 1 GiB. The rest run on the CPU from system memory. It never shrinks the context to make room, because Ollama set it.
On a large GPU the usual trigger is the default context. With 47 GiB or more of GPU memory the default is 262,144 tokens, capped at each model's trained context, and for llama3.3:70b that is 131,072 tokens and 40.0 GiB of KV cache next to a 43 GB download (our calculation). The second trigger is another process on the same GPU. nvidia-smi lists it, and the gpu memory line shows how little was free before the load.
The fixes, in the order I would try them:
- Lower the context to what the work needs, with
OLLAMA_CONTEXT_LENGTHon the server. Ollama context length has the three ways to set it and the cost per token for common models. - Set
OLLAMA_KV_CACHE_TYPE=q8_0, which roughly halves the cache for every model on the server. - Free the GPU: stop the other process, or unload other models with
ollama stopand the model's name. - Pull a smaller quantization of the model, or a smaller model. Ollama GPU requirements lists the sizes.
- Move to a GPU with more memory. The LLM VRAM calculator shows which one holds your model and context.
| GPU | Memory | From | Available now |
|---|---|---|---|
| RTX A6000 | 48 GB | $0.48/GPU-hr | Yes |
| A100 SXM4 80GB | 80 GB | $1.50/GPU-hr | Yes |
| RTX PRO 6000 Blackwell | 96 GB | $2.16/GPU-hr | Yes |
| H200 NVL | - | Not listed | No |
Prices checked 14 Nov 2026, 09:38 UTC
If the load itself runs out of memory and you never set a context, Ollama retries once with a smaller automatic one and logs the warning llama-server load failed; reducing automatic context and retrying once. That is why a model can come up with 32,768 or 4,096 tokens in the CONTEXT column when you expected more.
The driver changed under the running system
A driver update that lands while the old kernel module is still loaded leaves the two halves of the driver at different versions, and no new process can open the GPU until they match. The sign is on the host and has nothing to do with Ollama: nvidia-smi stops working and reports a driver and library version mismatch, and the kernel log has a line that begins NVRM: API mismatch. On Ubuntu this can happen without anyone running apt, because the unattended-upgrades package is installed by default and applies security updates automatically.
An Ollama server that starts in that state finds no GPU and logs inference compute with library=cpu. Ollama's troubleshooting page lists the checks: load the UVM driver with sudo nvidia-modprobe -u, reload it with sudo rmmod nvidia_uvm and sudo modprobe nvidia_uvm, reboot, and run the latest driver, then read the kernel log with sudo dmesg | grep -i nvrm if none of that helps. For a version mismatch the reboot is what loads the new module. Then restart Ollama and read the startup line again. On QuantaCloud a new instance always starts from the same clean image, which is the other way out when the instance holds nothing you still need.
Installing NVIDIA drivers on Ubuntu shows how to hold the driver packages so this cannot happen in the middle of a job, and checking the driver and CUDA version explains what nvidia-smi prints when it works.
The container was started without the GPU
A container gets a GPU only if you ask for one when you create it. Without --gpus=all on docker run, or the device reservation in a Compose file, the official image starts normally, serves the API and runs every model on the CPU. One command settles it:
sudo docker exec ollama nvidia-smi
If it prints the GPU table, the container has the GPU. If the command is not found, the container was created without it: remove the container and create it again with the flag. The models stay in the volume if you mounted one. If --gpus=all itself fails, the host is missing the NVIDIA Container Toolkit, and Ollama's own test for that is docker run --gpus all ubuntu nvidia-smi. Ollama in Docker has the full run command, and Docker with NVIDIA GPUs covers installing the toolkit.
There is one more Docker case: a container that used the GPU and later fell back to the CPU. Ollama's troubleshooting page and NVIDIA's both describe it. On hosts where systemd manages the container's cgroups, a systemctl daemon-reload on the host can take the GPU away from a running container, and nvidia-smi inside it then answers Failed to initialize NVML: Unknown Error. Deleting and recreating the container brings the GPU back. The lasting fix both pages give is to set "exec-opts": ["native.cgroupdriver=cgroupfs"] in /etc/docker/daemon.json and restart Docker.
An environment variable hides the GPU
Two variables can make a working GPU invisible to Ollama, and both tend to be left over from an earlier experiment.
CUDA_VISIBLE_DEVICES limits which NVIDIA GPUs a process sees. Ollama's docs note that an invalid ID such as -1 forces the CPU, and an index that does not exist on this machine does the same. When the variable is set at all, 0.40.16 logs the warning user overrode visible devices at startup, followed by if GPUs are not correctly discovered, unset and try again. The CUDA_VISIBLE_DEVICES guide explains the numbering.
OLLAMA_LLM_LIBRARY tells Ollama to skip autodetection and use one named build. In 0.40.16 the NVIDIA builds are called cuda_v12 and cuda_v13. A value copied from an older guide, such as cpu_avx2, matches neither, so discovery skips both and the server stays on the CPU.
The server config line in the log shows the value of both as the server read them. Look for where they were set: systemctl cat ollama prints the service with its overrides, and for a container the variables are in the docker run command or the Compose file. Remove the line and restart the server.
The GPU or its driver is too old
Ollama bundles its own CUDA builds, so it does not need CUDA installed on the host. It needs a driver that is new enough for those builds: version 550 or newer for GPUs with compute capability 5.0 and up, and 570 or newer for capability 5.0 through 6.2. Release 0.40.16 ships two builds, for CUDA 12.8 and CUDA 13.0, and the libdirs field of the startup line names the one in use.
When the driver is too old, the log has the warning NVIDIA driver too old with a required_driver field of 550 or newer or 570 or newer. When the GPU itself is outside what the build was compiled for, the log has a line that ends compute capability not in compiled architectures. Either way that GPU is dropped and Ollama carries on without it.
On QuantaCloud this cause is unlikely, because every GPU in the catalog has compute capability 8.0 or higher. That leaves the driver, which one command reads:
nvidia-smi --query-gpu=driver_version --format=csv,noheader
A parameter limits the layers
num_gpu is the number of layers Ollama sends to the GPU. It is automatic by default, and a model, a request or a client can set it. num_gpu 0 means CPU only: Ollama does not even offer the GPU to that model. Any other number caps the layers at that number.
It gets set in three places. A Modelfile can carry PARAMETER num_gpu, which ollama show --modelfile prints when you give it the model's name. An API request can send it in options. And a chat client can send it for you: in Open WebUI 0.13.3, switching on the num_gpu control in the advanced parameters pre-fills it with 0, which is CPU only. In the log, the starting llama-server line of that model then contains -ngl followed by the number.
The fix is to remove it, not to raise it. Create the model again without the parameter, or switch the control off in the client, and let Ollama choose.
There is no switch that forces the GPU#
Ollama already puts every layer on the GPU that fits, so there is nothing to switch on. What looks like a force option, num_gpu set to a number at or above the model's layer count, only changes how a model that does not fit fails. With the layer count set by you, llama.cpp stops fitting the model to free memory, so the load fails instead of splitting, or succeeds only after Ollama's one retry with a smaller automatic context. It never makes a model fit that did not fit before.
So leave num_gpu automatic and make the model fit. A shorter context costs nothing, and it is the first fix to try. I would also not chase utilization figures before the PROCESSOR column reads 100% GPU. Low GPU utilization and CUDA out of memory cover the training side of both problems.
Ollama and GPU FAQ#
Does Ollama need CUDA installed?
No. Ollama ships its own CUDA builds and needs only the NVIDIA driver on the host, version 550 or newer. In Docker the host also needs the NVIDIA Container Toolkit, and the container needs --gpus=all.
Which GPUs does Ollama support?
NVIDIA GPUs with compute capability 5.0 or newer, according to its hardware page. AMD GPUs run through ROCm 7, and other GPUs through Vulkan. Every GPU QuantaCloud rents is an NVIDIA GPU with compute capability 8.0 or higher.
How do I make Ollama use the GPU?
You do not have to: it uses the GPU whenever it finds one and the model fits. If it does not, read the PROCESSOR column of ollama ps and the inference compute line of the log, and work down the causes above.
How do I run Ollama on the CPU only?
Set CUDA_VISIBLE_DEVICES=-1 on the server, which Ollama's docs give as the way to ignore the GPUs, or set num_gpu to 0 for one model.
Why does ollama ps show nothing?
No model is loaded. Ollama unloads a model 5 minutes after its last request by default, so send a request first, or raise OLLAMA_KEEP_ALIVE.
My order of operations: ollama ps, then the inference compute line, then the six causes from the top. On a GPU server the first one is where I would look before anything else: a context larger than the card, fixed by a number in OLLAMA_CONTEXT_LENGTH. When the context you need really is larger than the card, move up a tier: the 96 GB RTX PRO 6000 holds llama3.3:70b with its full context, and Open WebUI on QuantaCloud has the model table for the rest.