Install or update NVIDIA drivers on an Ubuntu GPU server

The way I install an NVIDIA driver on an Ubuntu server is from packages, never from NVIDIA's run file: Ubuntu's own -server packages when the branch they carry is new enough, and NVIDIA's apt repository when it is not. On every GPU from Turing on, choose the open kernel modules, and on Blackwell GPUs they are the only kind that works. Then reboot, because the kernel module and the libraries must be the same version. On a QuantaCloud instance the driver is already installed on every template, so the last section says when it is still worth changing, and checking your driver and CUDA version shows how to read what you have.

Which way to install#

The choice is between three sources, and on a server two of them are sound:

The run file is the wrong choice on a server for the reason NVIDIA gives: it does not update with the distribution's package manager, and NVIDIA recommends the distribution packages where possible. Do not mix the sources either. Ubuntu warns that a driver from elsewhere can overwrite the one from ubuntu-drivers and break Secure Boot, so remove one before you install the other.

Canonical's -server packages are the Enterprise Ready Drivers, which follow NVIDIA's data center driver releases. NVIDIA supports three branches today: 580, a long term support branch until June 2028, 595, a production branch until March 2027, and 620, a long term support branch until September 2029. That makes 580-server-open the branch I pick from Ubuntu's archive.

Open or proprietary kernel modules#

The open kernel modules are the right choice on any GPU QuantaCloud offers. NVIDIA recommends them on every GPU that supports them, Blackwell and later work only with them, and they cannot drive GPUs older than Turing. From branch 615, NVIDIA's own packages ship only the open modules. In Ubuntu's archive, the open ones are the packages that end in -open.

To see which kind is loaded:

modinfo nvidia | grep license        # Dual MIT/GPL means the open modules
cat /proc/driver/nvidia/version      # "Open Kernel Module" in the first line

Which driver branch each GPU needs#

The three supported branches, 580, 595 and 620, list every GPU in the table. Older branches stop short, which matters on an image that still runs one:

Volta, Pascal and Maxwell cards such as the V100, P100 and M60 end at 580. From 590 on, NVIDIA lists them as legacy GPUs served by the 580 driver, and they need the proprietary modules. The CUDA side adds its own floor: builds made with CUDA 13 need driver 580 or newer.

The driver is not the CUDA toolkit#

The driver is what every GPU program needs: the kernel module, the CUDA driver library and nvidia-smi. The CUDA toolkit is the compiler and the headers and libraries for building CUDA code, and most servers never need it, because PyTorch wheels and CUDA containers bring their own CUDA runtime. Install it only when something compiles CUDA code on the machine, from NVIDIA's repository with the keyring from the steps below:

sudo apt install cuda-toolkit-13-5

cuda-toolkit-13-5 stays on CUDA 13.5, the release of 2026-11-04 that belongs to branch R620, and existing CUDA 13 programs also run on 580 and newer. Since CUDA 13.4, none of the toolkit meta packages installs or depends on a driver. Checking the driver and CUDA version shows how to match a toolkit to the driver you have.

Install with Ubuntu's packages#

These steps are the same on Ubuntu 22.04 and 24.04, which both carry 580-server-open at 580.178.04, the release NVIDIA published on 2026-08-03:

1. Bring the system and kernel up to date, then reboot:

   TerminalCopy

   sudo apt update && sudo apt upgrade
   sudo reboot
   

2. List the server drivers Ubuntu offers for your GPU. If the command is missing, install the ubuntu-drivers-common package:

   TerminalCopy

   sudo ubuntu-drivers list --gpgpu
   

3. Install the 580 open server driver and its tools, which include nvidia-smi:

   TerminalCopy

   sudo ubuntu-drivers install --gpgpu nvidia:580-server-open
   sudo apt install nvidia-utils-580-server
   

4. Reboot and check:

   TerminalCopy

   sudo reboot
   nvidia-smi
   

ubuntu-drivers installs Canonical's precompiled, signed modules, so this path also works with Secure Boot on.

Install from NVIDIA's repository#

NVIDIA's repository is the way to a branch Ubuntu does not carry, such as 620. On Ubuntu 24.04, replace ubuntu2204 with ubuntu2404 in the URL:

1. Install the headers for the running kernel, which DKMS needs to build the module:

   TerminalCopy

   sudo apt install linux-headers-$(uname -r)
   

2. Add NVIDIA's repository:

   TerminalCopy

   wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
   sudo dpkg -i cuda-keyring_1.1-1_all.deb
   sudo apt update
   

3. Choose the branch before you install the driver. Without a pinning package, apt follows the newest branch:

   TerminalCopy

   sudo apt install nvidia-driver-pinning-620
   

4. Install the driver with the open modules, or only the compute parts on a headless server. Both include nvidia-smi:

   TerminalCopy

   sudo apt install nvidia-open
   # or, without the desktop components:
   sudo apt -V install libnvidia-compute nvidia-dkms-open
   

5. Reboot and check that the loaded module is the one you installed:

   TerminalCopy

   sudo reboot
   cat /proc/driver/nvidia/version
   nvidia-smi
   

From branch 590, NVIDIA's Ubuntu packages carry no branch number in their names, so the pinning package decides the branch. To upgrade later, run sudo apt update && sudo apt dist-upgrade and reboot.

When an update breaks nvidia-smi#

The error everyone meets after an update is this one:

Failed to initialize NVML: Driver/library version mismatch

It means the driver's libraries on disk were replaced while the old kernel module stayed loaded, and sudo dmesg shows both versions in a line that starts with NVRM: API mismatch: the client has the version. On Ubuntu's packages this happens on its own: Ubuntu publishes driver updates in the -security pocket, and unattended-upgrades installs security updates once a day by default. The fix is a reboot, which loads the new module. Do it between jobs.

To keep it from happening under a long job, take driver updates on your schedule. For Ubuntu's packages, add three entries to the Unattended-Upgrade::Package-Blacklist block that already exists in /etc/apt/apt.conf.d/50unattended-upgrades, so that it reads:

Unattended-Upgrade::Package-Blacklist {
    "linux-";
    "nvidia-";
    "libnvidia-";
};

The kernel is on the list on purpose. Each precompiled module package in Ubuntu's archive depends on one exact driver version, so the kernel and the driver have to move together, and Ubuntu notes that blocking a package can hold back the updates that depend on it. The price is that kernel security updates stop installing themselves too, so run sudo apt update && sudo apt upgrade by hand between jobs and reboot. sudo apt-mark hold followed by package names stops manual upgrades of those packages too, and apt-mark showhold lists what is held.

NVIDIA's repository behaves differently. Unattended-upgrades installs only from Ubuntu's own origins by default, and NVIDIA's repository identifies itself as Origin: NVIDIA, so its driver changes only when you run apt. To freeze one version, install its pinning package, for example sudo apt install nvidia-driver-pinning-620.70.06. DKMS then rebuilds that same driver for each new kernel, and dkms status shows whether the build for the new kernel succeeded before you reboot into it.

Remove a driver cleanly#

Removal follows the source the driver came from. For Ubuntu's packages, name the branch:

sudo apt --purge remove '*nvidia*580*'
sudo apt autoremove

For NVIDIA's packages, this is the command from NVIDIA's guide for Ubuntu 22.04 and 24.04:

sudo apt remove --autoremove --purge -V \
  cuda-compat\* libnvidia-cfg1\* libnvidia-compute\* libnvidia-decode\* \
  libnvidia-encode\* libnvidia-extra\* libnvidia-fbc1\* libnvidia-gl\* \
  libnvidia-gpucomp\* libnvidia-nscq\* libnvsdm\* libxnvctrl\* \
  nvidia-dkms-open\* nvidia-driver\* nvidia-fabricmanager\* nvidia-firmware\* \
  nvidia-headless-open\* nvidia-headless-no-dkms-open\* nvidia-imex\* \
  nvidia-kernel-common\* nvidia-kernel-source-open\* nvidia-modprobe\* \
  nvidia-open\* nvidia-persistenced\* nvidia-settings\* nvidia-xconfig\* \
  xserver-xorg-video-nvidia\*

A driver from the run file comes out with sudo nvidia-uninstall. Reboot after any of the three. If you are not sure what installed the current driver, dpkg -l | grep -i nvidia lists the packages and their versions.

Secure Boot#

Secure Boot only lets the kernel load signed modules, so check it before you choose a source:

mokutil --sb-state

With Secure Boot on, Ubuntu's precompiled modules are the easy path, because Canonical signs them. DKMS modules, from NVIDIA's repository or Ubuntu's nvidia-dkms packages, are not signed with Canonical's key: installing them can walk you through creating and enrolling your own key, and an unsigned module simply does not load. With Secure Boot off, both paths work.

No devices were found, and other driver errors#

Each failure points to a different layer:

The kernel log names the reason when the module fails to load, and NVIDIA's README suggests loading it by hand to see the error:

sudo dmesg | grep -i nvrm
sudo modprobe nvidia

Ubuntu's fix for Nouveau, followed by a reboot:

echo "blacklist nouveau" | sudo tee /etc/modprobe.d/disable-nouveau.conf
echo "options nouveau modeset=0" | sudo tee -a /etc/modprobe.d/disable-nouveau.conf
sudo update-initramfs -u

The nvidia-smi commands guide covers what the tool reports once the driver works.

On a QuantaCloud instance#

The driver comes preinstalled, so on QuantaCloud I fit the software to the driver instead of the other way round. Every template runs on the same Ubuntu 22.04 image with the NVIDIA driver and Docker, and the image does not pin a version, so read it with nvidia-smi on your own instance. The case for changing it is a stack that needs a newer branch than the image has: CUDA 13 builds need 580 or newer, and CUDA 13.5 features need R620. Most of the time a PyTorch wheel or container image built for the installed driver is the cheaper answer, as the CUDA version guide and Docker with a GPU show.

Two facts make a driver change costly here. It needs a reboot, and nobody has tested yet whether a reboot from inside a QuantaCloud VM brings it back. And stopping an instance deletes its disk, so the new driver has to be installed again on every new instance. If you still need it, put the steps in a setup script, run it right after launch, and find out what installed the current driver with dpkg -l | grep -i nvidia before you replace it.

My rule for a GPU server: Ubuntu's 580-server-open packages unless I need a newer branch, NVIDIA's repository with a pinning package when I do, the open modules on everything from Turing on, and a reboot after every driver change, scheduled between jobs. On QuantaCloud the driver is already there, so start with nvidia-smi and choose PyTorch and images to fit it, and pick the GPU from the catalog.