CUDA_VISIBLE_DEVICES: choose which GPUs a process can use

CUDA_VISIBLE_DEVICES is a list of the GPUs a process may use. CUDA hides every GPU that is not on the list and numbers the rest from 0, in the order you wrote them, so CUDA_VISIBLE_DEVICES=1 python train.py runs the job on the second GPU and the job calls it cuda:0. It has to be set before the program starts using CUDA, which in practice means on the command line or in the environment of the service. On a multi-GPU QuantaCloud VM it is the simplest way to give each job its own GPU.

What the variable does#

NVIDIA's CUDA documentation defines every value it accepts:

The program never sees the hidden GPUs. CUDA's device count includes only the visible ones, so with 2,1 a call that selects device 0 gets the GPU that nvidia-smi lists as 2.

Why the numbers disagree with nvidia-smi#

The first difference is the renumbering, and it is always there. With CUDA_VISIBLE_DEVICES=2,3, PyTorch's cuda:0 and cuda:1 are the GPUs that nvidia-smi shows as 2 and 3.

The second is the order. CUDA's default, CUDA_DEVICE_ORDER=FASTEST_FIRST, numbers the GPUs from fastest to slowest by a simple heuristic, while nvidia-smi lists them by its own index, and NVIDIA does not promise that index stays the same across reboots. CUDA_DEVICE_ORDER=PCI_BUS_ID makes CUDA count in ascending PCI bus ID, which you can compare with the bus IDs nvidia-smi prints:

nvidia-smi --query-gpu=index,name,pci.bus_id,uuid --format=csv
export CUDA_DEVICE_ORDER=PCI_BUS_ID

My rule is to set CUDA_DEVICE_ORDER=PCI_BUS_ID whenever I select GPUs by number, and to use UUIDs in anything that has to survive a reboot.

Select GPUs by UUID#

A UUID names the same GPU everywhere, and NVIDIA recommends it, or the PCI bus ID, for anything that must stay consistent. nvidia-smi -L prints one line per GPU with its UUID, and the first characters are enough when they are unique:

nvidia-smi -L
CUDA_VISIBLE_DEVICES=GPU-8932f937 python train.py

To check which physical GPU a PyTorch process got, print the UUID of each visible device and compare it with nvidia-smi -L. PyTorch prints the UUID without the GPU- prefix, so the script adds it:

import torch

for i in range(torch.cuda.device_count()):
    p = torch.cuda.get_device_properties(i)
    print(f"cuda:{i}", p.name, f"GPU-{p.uuid}")

Choose a GPU inside PyTorch#

PyTorch's own advice is to prefer CUDA_VISIBLE_DEVICES in most cases, and to use torch.device or the torch.cuda.device context manager when one process works on several GPUs:

model = model.to("cuda:1")            # the second visible GPU

with torch.cuda.device(1):
    x = torch.randn(1024, 1024, device="cuda")   # also on cuda:1

To set the variable from Python, set it before import torch and before anything else that uses CUDA. PyTorch caches the device count once CUDA has initialized, and changing the variable after that has no effect.

One job per GPU on a multi-GPU VM#

The pattern for independent jobs, such as hyperparameter runs or batch inference shards, is one process per GPU, each with its own value. This loop reads the UUIDs from nvidia-smi, starts one job per GPU in the background and waits for all of them:

mapfile -t GPUS < <(nvidia-smi --query-gpu=uuid --format=csv,noheader)
for i in "${!GPUS[@]}"; do
  CUDA_VISIBLE_DEVICES=${GPUS[$i]} python train.py --shard "$i" > "run-$i.log" 2>&1 &
done
wait

Save it as a script and run it with bash. Each job sees exactly one GPU as cuda:0, so the training script needs no device logic of its own. The logs sit on the instance's disk, and stopping a QuantaCloud instance deletes that disk, so copy them off first. One process across all the GPUs is a different setup, covered in PyTorch DDP and FSDP.

The Docker equivalents#

Docker selects GPUs with --gpus, using the index nvidia-smi shows, and the container sees only those:

Inside the container only the GPUs you passed exist, so the program numbers them from 0 again. Running Docker with a GPU covers the NVIDIA Container Toolkit these flags rely on.

How vLLM, Ollama and ComfyUI choose devices#

All three accept the variable, and two have options of their own:

Ollama usually runs as a systemd service, which does not see variables from your shell. Add the variable to the service instead:

sudo systemctl edit ollama.service
# add under [Service]:  Environment="CUDA_VISIBLE_DEVICES=GPU-8932f937"
sudo systemctl daemon-reload
sudo systemctl restart ollama

For vLLM, two servers on two GPUs need two ports, and deploying vLLM with Docker has the full command.

Mistakes that select the wrong GPU#

Most of these fail silently, so check the UUIDs whenever a job lands somewhere unexpected:

The nvidia-smi commands guide has more on reading indices, UUIDs and bus IDs.

My rule: one GPU per process with CUDA_VISIBLE_DEVICES set on the command line, UUIDs in scripts, and CUDA_DEVICE_ORDER=PCI_BUS_ID whenever I use numbers. It works the same on a 2-GPU VM as on an 8-GPU one, so start with two GPUs and move up when the catalog lists the larger configuration you need, or pick another GPU from the catalog.