Run Ollama in Docker with an NVIDIA GPU

Running Ollama in Docker on an NVIDIA GPU takes one command: start the official ollama/ollama image with --gpus=all, a volume for the models, and port 11434 published on 127.0.0.1. The flag hands the GPUs to the container, the volume keeps your models when the container is replaced, and the loopback address keeps an API that has no login off the internet. Two things go wrong in practice: a host without the NVIDIA Container Toolkit, where --gpus fails, and a port published on every interface, where anyone who finds it can use your GPU. This guide uses image tag 0.40.16, pushed on 2026-11-09.

The official image#

The image is Ollama's own build of its server for Linux, with the GPU libraries inside.

Pin the version tag. Ollama published 13 stable releases in October 2026 alone, so a container created from latest can be a different server from one week to the next.

Start the container#

The host needs the NVIDIA driver, Docker and the NVIDIA Container Toolkit. QuantaCloud's Bare Metal template is an Ubuntu 22.04 VM with the driver and Docker already installed, and the deploy docs cover launching it.

1. Connect as ubuntu and check that Docker can hand a GPU to a container:

   TerminalCopy

   ssh -i ~/.ssh/your_key ubuntu@<instance-ip>
   nvidia-smi
   sudo docker run --rm --gpus all ubuntu nvidia-smi
   

   If the last command prints the same GPU table as the one before it, the toolkit is in place. If Docker answers with an error, fix that first with the section on what goes wrong below.

2. Start Ollama with a pinned tag:

   TerminalCopy

   sudo docker run -d --gpus=all --restart unless-stopped \
     -v ollama:/root/.ollama \
     -p 127.0.0.1:11434:11434 \
     --name ollama ollama/ollama:0.40.16
   

3. Pull a model, talk to it, and check the API from the VM. Type /bye to leave the chat:

   TerminalCopy

   sudo docker exec ollama ollama pull gpt-oss:20b
   sudo docker exec -it ollama ollama run gpt-oss:20b
   curl http://127.0.0.1:11434/api/version
   

Each part of the run command has a job:

gpt-oss:20b is a 14 GB download and fits a 48 GB RTX A6000 with room for a long context. Ollama GPU requirements lists what fits on 48, 80, 96 and 141 GB.

Prove the container sees the GPU#

A container that lost its GPU still answers, only slowly, so check it once instead of assuming. Three commands do it:

sudo docker exec ollama nvidia-smi
sudo docker logs ollama 2>&1 | grep "inference compute"
sudo docker exec ollama ollama ps

The first prints the same GPU table as on the host, which proves the toolkit passed the GPU through. The second shows the line Ollama logged for each GPU it found at startup: it should say library=CUDA with the GPU's name and memory, and library=cpu means it found none. The third, run after the model has answered once, lists the loaded model, and its PROCESSOR column should read 100% GPU.

A CPU share in that column while the log shows a GPU means the model and its context do not fit in GPU memory. Ollama not using the GPU goes through that and the other causes in order.

Environment variables that matter in a container#

Of Ollama's own variables the image sets one, OLLAMA_HOST=0.0.0.0:11434, so that the published port can reach the server inside the container. Leave it alone. The ones worth adding are the same as on any GPU server, passed with -e when you create the container:

sudo docker run -d --gpus=all --restart unless-stopped \
  -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 \
  -e OLLAMA_CONTEXT_LENGTH=32768 -e OLLAMA_KEEP_ALIVE=-1 \
  --name ollama ollama/ollama:0.40.16

A container's environment is fixed when it is created, so changing a variable means removing the container with sudo docker rm -f ollama and running it again. The models stay in the volume. To check what the server read, look for the server config line with sudo docker logs ollama 2>&1 | grep "server config". To give the container one GPU of several, use Docker's own flag, --gpus device=0, in place of --gpus=all. Ollama environment variables has every setting with its default, and Ollama context length explains what the first one costs in memory.

A Compose file with Open WebUI#

Compose is the same container written down, plus a chat interface next to it. Use it in place of the single container: if that one is still running, remove it first with sudo docker rm -f ollama.

1. Create a folder and a fixed secret key for Open WebUI. Without one, Open WebUI generates a key inside its container, and every recreated container logs everyone out.

   TerminalCopy

   mkdir -p ~/ollama && cd ~/ollama
   echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)" > .env
   

2. Write compose.yaml in that folder:

   YAMLCopy

   services:
     ollama:
       image: ollama/ollama:0.40.16
       container_name: ollama
       restart: unless-stopped
       ports:
         - "127.0.0.1:11434:11434"
       volumes:
         - ollama:/root/.ollama
       environment:
         - OLLAMA_CONTEXT_LENGTH=32768
         - OLLAMA_KEEP_ALIVE=-1
       deploy:
         resources:
           reservations:
             devices:
               - driver: nvidia
                 count: all
                 capabilities: [gpu]
   
     open-webui:
       image: ghcr.io/open-webui/open-webui:v0.13.3
       container_name: open-webui
       restart: unless-stopped
       depends_on:
         - ollama
       ports:
         - "127.0.0.1:3000:8080"
       volumes:
         - open-webui:/app/backend/data
       environment:
         - OLLAMA_BASE_URL=http://ollama:11434
         - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
         - ENABLE_OPENAI_API=false
         - ENABLE_VERSION_UPDATE_CHECK=false
   
   volumes:
     ollama:
     open-webui:
   

3. Start both and pull a model:

   TerminalCopy

   sudo docker compose up -d
   sudo docker compose ps
   sudo docker exec ollama ollama pull gpt-oss:20b
   

4. From your laptop, forward both ports over SSH, open http://localhost:3000 and create the admin account straight away:

   TerminalCopy

   ssh -N -L 3000:127.0.0.1:3000 -L 11434:127.0.0.1:11434 \
     -i ~/.ssh/your_key ubuntu@<instance-ip>
   

The deploy block is Compose's form of --gpus=all, and its capabilities line is required. Open WebUI reaches Ollama over the Compose network at http://ollama:11434, so the ports entry on the ollama service is there only for your own API clients: delete it if nothing but Open WebUI talks to Ollama. Both tags are pinned. Open WebUI's main and latest tags pointed at a different image from v0.13.3 on 2026-11-12, and v0.13.3, released on 2026-11-08, was the newest release. The first account created on a fresh Open WebUI becomes its administrator and sign-up then closes, which is the reason not to leave the page unclaimed.

Compose keeps its own volumes, named after the folder, so models you pulled into the single container's ollama volume are not in this one. The Open WebUI setup guide covers the interface itself: settings, more users and saving your chats.

Update the image#

An update is a new tag and a new container, and the models stay where they are. Read the release notes first, then, for the single container:

NEW=0.40.16   # the version you are moving to
sudo docker pull ollama/ollama:$NEW
sudo docker rm -f ollama
sudo docker run -d --gpus=all --restart unless-stopped \
  -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama:$NEW
curl http://127.0.0.1:11434/api/version

With Compose, change the tag in compose.yaml, then run sudo docker compose pull and sudo docker compose up -d. Compose recreates a container whose image or configuration changed and keeps its volumes.

One thing to know when a volume is older than the image: the release notes of 0.40.2 say that models downloaded with earlier versions of Ollama are upgraded in the background the first time they run, and that the original copy stays on disk as a backup. On a fresh instance, where the version you run pulled every model, that does not come up.

The two things that go wrong#

Docker cannot find the GPU

The first failure is a host without the NVIDIA Container Toolkit, or one where Docker was not restarted after installing it. --gpus=all then fails before the container starts, with could not select device driver "" with capabilities: [[gpu]], or on recent Docker releases with failed to discover GPU vendor from CDI. The fix is the install from Ollama's own Docker page:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then run sudo docker run --rm --gpus all ubuntu nvidia-smi again. Docker with NVIDIA GPUs has the same install with a pinned toolkit version, and the other errors Docker prints.

The quieter form of the same mistake is a container created without --gpus=all, or a Compose service without the deploy block. Nothing fails. Ollama starts, answers, and runs every model on the CPU, which is why the three checks above are worth the minute.

Port 11434 is open to the internet

The second failure works perfectly, which is the problem. Ollama's own example publishes the port with -p 11434:11434, and Docker's documentation is blunt about what that does: publishing a port makes it available to the outside world as well as to the host, and the traffic is routed before ufw's rules apply, so the firewall you set up does not help. Ollama's local API has no authentication. Anyone who reaches port 11434 can run your models, pull new ones and delete the ones you have.

Check how your container is published:

sudo docker port ollama

11434/tcp -> 127.0.0.1:11434 is what you want. 0.0.0.0:11434 means every interface, the public one included: remove the container and create it again with the 127.0.0.1: prefix. Docker releases older than 28.0.0 also let other hosts on the same network segment reach ports published to localhost, so check sudo docker version as well.

To use the API from your laptop, tunnel to it instead of opening it:

ssh -N -L 11434:127.0.0.1:11434 -i ~/.ssh/your_key ubuntu@<instance-ip>

After that http://localhost:11434 on your laptop is the Ollama on the GPU. The Ollama API guide covers the tunnel, the Python and OpenAI clients, and a token-checking HTTPS proxy for the case where an app has to reach it without you.

The template gives you this already built#

QuantaCloud's Open WebUI + Ollama template starts Ollama and Open WebUI in a container at launch, with nothing to type. You open the chat interface from the Deployments page, behind your QuantaCloud login, pull a model and start. Open WebUI on QuantaCloud describes it.

I would still run it myself in four cases:

Before you stop the instance#

Docker volumes live on the instance's disk, and stopping a QuantaCloud instance terminates it and deletes that disk. The models, the Open WebUI accounts and the chats go with it, and there are no snapshots to bring them back. Keep compose.yaml and a list of the models you pull on your laptop, so the next instance is two commands away, and export your chats first if you used Open WebUI. Billing stops when you stop, and the unused seconds of the current hour are refunded, as the pricing page explains.

Ollama in Docker FAQ#

Does the Ollama Docker image need CUDA on the host?

No. The image carries Ollama's own CUDA builds. The host needs the NVIDIA driver and the NVIDIA Container Toolkit, and the container needs --gpus=all.

Where does the container store models?

Under /root/.ollama inside the container, which the run command maps to the Docker volume named ollama. sudo docker volume inspect ollama shows where that volume sits on the host's disk.

Can I use the latest tag?

It works, and it changes under you: 13 stable releases went out in October 2026. Pin a version tag and move it when you choose to.

How do I run ollama commands when it is in a container?

Prefix them with sudo docker exec -it ollama. For example, sudo docker exec -it ollama ollama list shows the models in the volume.

How do I give the container only one GPU?

Use --gpus device=0 in place of --gpus=all. In Compose, replace count: all with device_ids: ['0'].

My rule for Ollama in Docker: pin the tag, mount the volume, publish on 127.0.0.1, and run the three GPU checks before trusting any speed you see. Use the template when all you want is to chat, and your own container when you want the versions, the settings or the API under your control. A 48 GB RTX A6000 on the Bare Metal template is where I would start.