The handful of nvidia-smi commands worth knowing on a GPU server are the plain table, the CSV query in a loop, dmon and pmon for a rolling view, and -q -d for detail on memory, ECC, power and temperature. nvidia-smi comes with the NVIDIA driver, and QuantaCloud's Bare Metal template is an Ubuntu 22.04 VM with the driver and Docker in place, so there is nothing to install first. Every option below is in NVIDIA's manual for nvidia-smi 620.70, the release that shipped with driver 620.70.06 on 2026-11-04, and every query field used here is also present in the 535, 580 and 595 builds. What the driver and CUDA numbers in the header mean is a separate question, answered in checking your driver and CUDA version.
Read the default table#
The default output is two tables: two lines per GPU, then the processes that hold memory on them. This sample comes from Docker's GPU documentation, an NVIDIA L4 on driver 535.288.01:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.288.01 Driver Version: 535.288.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA L4 Off | 00000000:31:00.0 Off | 0 |
| N/A 40C P0 27W / 72W | 0MiB / 23034MiB | 4% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
| Field | What it shows | What to look for |
|---|---|---|
| GPU | The index, starting at 0 | NVIDIA does not keep the order fixed between reboots, so scripts should use the UUID |
| Name | The product name | The GPU you are paying for |
| Persistence-M | Whether persistence mode is on | Off means the driver unloads when nothing uses the GPU |
| Bus-Id | The PCI address as domain:bus:device.function | The stable way to tell two identical GPUs apart |
| Disp.A | Whether a display is initialized on the GPU | Off on a headless server |
| Volatile Uncorr. ECC | Uncorrectable memory errors since the driver loaded | 0. Anything else belongs in the health check |
| Fan | The intended fan speed in percent | N/A on cards cooled by the chassis |
| Temp | The GPU temperature in degrees C | Compare with the thresholds from -q -d TEMPERATURE |
| Perf | The performance state, P0 (maximum) to P12 (minimum) | P0 is full performance |
| Pwr:Usage/Cap | Power draw and the power limit in watts | The cap is the limit that -pl changes |
| Memory-Usage | Used and total GPU memory in MiB | The total can sit below the nominal size, because ECC and the driver take a share |
| GPU-Util | The share of the sample period in which at least one kernel ran | Not how hard the GPU works |
| Compute M. | Default, exclusive process or prohibited | Default lets several processes share the GPU |
| MIG M. | Whether the GPU is split into MIG instances | N/A or Disabled on a whole GPU |
| Processes | GPU index, PID, type, name and memory per process | Type C is compute, G graphics, M an MPS client, O other |
Drivers up to the 595 branch print Driver Version and CUDA Version in the header. The 610, 615 and 620 branches print KMD Version and CUDA UMD Version for the same two values.
Loop the table with -l or watch#
The table refreshes in two ways: nvidia-smi can loop by itself, or watch can redraw it in place.
nvidia-smi -l 1 # a new table every second, until Ctrl+C
nvidia-smi -lms 500 # every 500 milliseconds
watch -n 1 nvidia-smi # redraw in place every second
watch -d -n 1 nvidia-smi # and highlight what changed
With -q or --query-gpu, -l without a number waits 5 seconds, and on Linux NVIDIA's manual says ECC and Xid events print during the wait, which a watch loop never shows. watch runs every 2 seconds unless -n says otherwise, and does not go below 0.1 seconds.
Query GPUs as CSV#
The form to use in scripts is --query-gpu, which prints only the fields you name, one line per GPU:
nvidia-smi --query-gpu=timestamp,index,name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \
--format=csv -l 5
--format=csv is mandatory. Add noheader to drop the first line and nounits to print bare numbers, and run nvidia-smi --help-query-gpu for the full list. These are the fields worth logging:
| Field | What it returns |
|---|---|
timestamp | The time of the query, as YYYY/MM/DD HH:MM:SS.msec |
index, uuid, pci.bus_id, name | Which GPU the line is about. The index can change at each boot, the UUID cannot |
utilization.gpu | Percent of the sample period in which a kernel was running |
utilization.memory | Percent of the sample period in which memory was read or written. It is not the share of memory in use |
memory.used, memory.free, memory.total | Memory allocated by active contexts, free memory and installed memory |
temperature.gpu | Core temperature in degrees C |
power.draw, power.limit | Board power in watts, averaged over one second on Ampere and newer, and the software power limit |
pstate | The performance state |
clocks_event_reasons.hw_thermal_slowdown, clocks_event_reasons.sw_thermal_slowdown | Whether the clocks are being held down because of temperature |
ecc.errors.uncorrected.volatile.total, ecc.errors.corrected.volatile.total | ECC errors since the driver loaded |
compute_mode, persistence_mode | The two settings described further down |
Query processes as CSV#
The process list has its own query, which is the quickest way to find what is holding memory:
nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_gpu_memory --format=csv
nvidia-smi --query-gpu=index,uuid --format=csv
The first command lists every process with a compute context and the memory that context holds. It names the GPU by UUID, so the second command gives you the index to match. A process you do not recognize is usually a notebook kernel or a server left running: end it with kill and its PID, and the memory comes back.
Rolling views with dmon and pmon#
The one-line-per-sample view is dmon for devices and pmon for processes. Both scroll, so the history stays on screen:
nvidia-smi dmon -s pucm -d 5 -o T # every 5 seconds, with the time on each line
nvidia-smi pmon -s um -c 10 # per-process figures, 10 samples, then exit
dmon -s letter | Columns it adds |
|---|---|
p | Power draw in watts, GPU and memory temperature in degrees C |
u | Utilization of the SMs, memory, encoder, decoder, JPEG and OFA engines in percent |
c | Processor and memory clocks in MHz |
v | Power violations in percent and thermal violations as a flag |
m | Frame buffer and BAR1 memory in use, in MB |
e | Aggregated single-bit and double-bit ECC errors, and PCIe replay errors |
t | PCIe receive and transmit throughput in MB/s |
Without -s, dmon shows the p, u and c groups, once a second, and from driver 550 on, --format csv turns its output into CSV for a log file. pmon takes u for utilization and m for memory, and its interval tops out at 10 seconds. On Hopper and newer GPUs, dmon --gpm-metrics 2,5,10 adds SM activity, tensor activity and DRAM activity. NVIDIA describes both commands for bare-metal Linux, and a metric the driver cannot read prints as a dash. Neither has been tested on a QuantaCloud VM yet, so if a column shows dashes, the CSV query above returns the same utilization and memory figures.
Detail for memory, ECC, power and temperature#
The full report is nvidia-smi -q, and -d cuts it down to the sections you ask for:
nvidia-smi -q -d MEMORY,ECC,ROW_REMAPPER -i 0
nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE
-d section | What it prints |
|---|---|
MEMORY | Frame buffer total, reserved, used and free, and BAR1 memory |
ECC | The ECC mode and the corrected and uncorrected error counts, volatile and aggregate |
ROW_REMAPPER | Rows remapped after memory errors, whether a remap is pending, and whether one has failed |
PAGE_RETIREMENT | Retired memory pages, the mechanism that row remapping replaced on Ampere and newer |
POWER | Average and instantaneous draw, and the current, default, minimum and maximum power limit |
TEMPERATURE | The current temperature and the shutdown, slowdown and maximum operating thresholds |
PERFORMANCE | The performance state and the reasons the clocks are being reduced, if any |
UTILIZATION, CLOCK, COMPUTE, PIDS | Utilization, clock frequencies, compute mode and the process list |
Volatile counters start again whenever the driver reloads, and aggregate counters are the lifetime of the card. What the ECC and row-remapping numbers mean for a server you rent is in the stress-test and health-check guide.
The topology matrix#
The command for how the GPUs in one machine reach each other is nvidia-smi topo -m. It prints a matrix with one cell per pair of devices, and each cell is one of these:
| Cell | The connection between the two GPUs |
|---|---|
NV# | A bonded set of that many NVLinks |
PIX | A single PCIe switch |
PXB | Several PCIe switches, without crossing the host bridge |
PHB | A PCIe host bridge, typically the CPU |
NODE | PCIe and the link between host bridges inside one NUMA node |
SYS | PCIe and the link between NUMA nodes |
X | The device itself |
The matrix also lists each GPU's CPU and NUMA affinity. On a VM it describes the topology the guest sees. The SXM, PCIe and NVLink guide explains why the difference matters for multi-GPU training.
Select a GPU with -i#
Every query and setting takes -i to act on one GPU, by index, UUID or PCI bus ID:
nvidia-smi -L
nvidia-smi -i 1
nvidia-smi -i GPU-3a23c669-1f69-c64e-cf85-44e9b07e7a2a --query-gpu=memory.used --format=csv
nvidia-smi -L prints one line per GPU with its UUID. NVIDIA recommends the UUID or the PCI bus ID over the index in anything that must keep working, because the enumeration order is not promised to stay the same between reboots. -i chooses which GPU nvidia-smi reports on or changes. It does not limit what a program can use: for that, set CUDA_VISIBLE_DEVICES.
Persistence mode, power limit and compute mode#
Three settings change how the GPU behaves, and all three need root:
| Command | What it does | How long it lasts |
|---|---|---|
sudo nvidia-smi -pm 1 | Turns on persistence mode: the driver stays loaded when no program uses the GPU, so the next CUDA program does not wait for it to initialize | Until the next reboot |
sudo nvidia-smi -pl 250 -i 0 | Caps the board power of GPU 0 at 250 W. The value must lie between the minimum and maximum power limit that nvidia-smi -q -d POWER prints | Until a reboot or a driver unload, and without persistence mode the driver unloads whenever no program uses the GPU |
sudo nvidia-smi -c 3 -i 0 | Sets compute mode to exclusive process, one context per GPU. -c 0 is the default, which lets several processes share it, and -c 2 prohibits compute use | Until the next reboot |
None of the three has been tested on a QuantaCloud VM, and a VM may refuse them. When a platform does not allow a change, nvidia-smi says so with a message such as Changing power management limit is not supported for GPU, Setting persistence mode is not supported for GPU ... on this platform or Setting compute mode to ... is not supported. The manual's return codes 3 and 4 mean that the operation is not available on the device and that the user has no permission for it. Read the settings before you plan around them:
nvidia-smi --query-gpu=index,persistence_mode,compute_mode,power.limit --format=csv
On a machine you install yourself, NVIDIA's distribution packages enable the nvidia-persistenced daemon, which starts with persistence mode on for every GPU, so the flag is not needed there. Installing NVIDIA drivers on Ubuntu covers those packages.
nvtop, nvitop and gpustat#
Three open-source tools read the same NVML library as nvidia-smi and add what it lacks: history graphs, an interactive process list and a one-line summary per GPU.
| Tool | Version checked | Install | What you get |
|---|---|---|---|
| nvtop | 3.4.0, released 2026-11-08 | sudo apt install nvtop gives 1.2.2 on Ubuntu 22.04 and 3.0.2 on 24.04. sudo snap install nvtop or the AppImage gives the current release | A top-style screen with utilization and memory graphs and a process list |
| nvitop | 1.7.2, released 2026-10-26 | uvx nvitop, pipx run nvitop or pip3 install --upgrade nvitop | A monitor with process filtering, a tree view and signals from the keyboard. nvitop -1 prints once, like nvidia-smi |
| gpustat | 1.1.1, released 2023-08-22 | pip install gpustat | One line per GPU. gpustat -cp --watch adds process names and PIDs and keeps refreshing |
All three are convenience on top of the driver. Anything you install disappears with the disk when a QuantaCloud instance stops, so I would add the one I use to a setup script and keep nvidia-smi as the tool every machine already has.
Log a training run to a file#
The simplest log is the CSV query appended to a file every few seconds while the job runs:
FIELDS=timestamp,index,utilization.gpu,utilization.memory,memory.used,temperature.gpu,power.draw
nvidia-smi --query-gpu=$FIELDS --format=csv > gpu-log.csv
while sleep 5; do
nvidia-smi --query-gpu=$FIELDS --format=csv,noheader >> gpu-log.csv
done &
LOGGER=$!
python train.py
kill $LOGGER
Save it as a script and run it with bash. The first call writes the header and the first sample, the loop adds one line per GPU every 5 seconds, and the last line stops the logger when training ends. The file answers the questions that come up after a run: the peak of memory.used says how much headroom the GPU had, and long stretches of low utilization.gpu are the pattern low GPU utilization explains. The file sits on the instance's disk, so copy it off before you stop the instance. The same query drives the watchdog in the auto-stop guide.
Why nvidia-smi and PyTorch disagree about memory#
The two tools count different things, so the numbers are not supposed to match:
| Number | Who reports it | What it counts |
|---|---|---|
Memory-Usage, memory.used | nvidia-smi | Everything all processes hold on the GPU |
used_gpu_memory per process | nvidia-smi | What one process's context holds, including PyTorch's cache and the CUDA context itself |
torch.cuda.memory_reserved() | PyTorch | Memory its caching allocator manages: live tensors plus blocks kept for reuse |
torch.cuda.memory_allocated() | PyTorch | Memory occupied by tensors only |
torch.cuda.mem_get_info() | CUDA, through PyTorch | Free and total memory of the whole GPU |
PyTorch keeps memory it has freed in a cache, and its documentation says that this unused memory still shows as used in nvidia-smi. So for one process the order is fixed: allocated is the smallest, reserved is the same or larger, and the figure in nvidia-smi is the largest, because it adds the CUDA context. torch.cuda.empty_cache() hands the cached blocks back and lowers the nvidia-smi figure, but it frees nothing a tensor still holds. When the job actually runs out, the numbers to read are in the error message, as fixing CUDA out of memory shows.
When nvidia-smi itself fails#
A failure of the command is a driver problem, and the message says which one:
| Message | What it means | Where to go |
|---|---|---|
nvidia-smi: command not found | The driver's tools are not installed, or a container was started without GPU access | Install the driver, or start the container with GPUs |
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver | The kernel module is not loaded, for example after a kernel update | The driver guide |
Failed to initialize NVML: Driver/library version mismatch | The driver's libraries were upgraded while the old kernel module is still loaded | The driver guide |
No devices were found | Nouveau holds the GPU, or your user cannot read the /dev/nvidia* device files | The driver guide |
GPU is lost, followed by the advice to reboot | The GPU has fallen off the PCIe bus or become inaccessible | The health-check guide |
My habit on a new instance is three commands: nvidia-smi once to see what I rented, nvidia-smi -q -d ECC,ROW_REMAPPER before anything long, and the CSV logger while the job runs. Everything else on this page is for the day one of those three shows something odd. Pick the GPU from the catalog, and keep the commands you use in the setup script that rebuilds your next instance.