Stress-test and health-check a GPU server on Linux

The check I run before a long job on any GPU server takes three steps: read the error counters and the kernel log, which takes seconds, then load every GPU for ten minutes with gpu-burn or NVIDIA's DCGM diagnostics, then run a short test with the framework the job will use. Each step catches failures the others miss. A GPU can pass a load test with uncorrectable memory errors already on record, and a clean log says nothing about how the card behaves at full power. On a rented VM, a failed check is a reason to stop the instance and report it, and the billing rules decide what that costs, as the last section explains.

The checks in order#

The order runs from cheap to expensive, and a failure early on saves you the rest:

Read the memory error counters#

The counters come first because they hold the GPU's own record of memory errors:

nvidia-smi -q -d ECC,ROW_REMAPPER

Row remapping is how Ampere and newer GPUs retire bad memory rows. A pending remapping takes effect only after a GPU reset, and NVIDIA's memory error guide treats a failed remapping, confirmed by its field diagnostic, as the point at which a GPU qualifies for return. Volatile counters start from zero whenever the driver reloads, so a clean volatile count on a fresh VM proves less than a clean aggregate count. The nvidia-smi commands guide explains the other sections of -q.

Xid errors in the kernel log#

The driver reports GPU errors to the kernel log as Xid messages, and NVIDIA's guide says to search for NVRM: Xid:

sudo dmesg | grep "NVRM: Xid"
sudo journalctl -k | grep "NVRM: Xid"     # the same, from the systemd journal

A line such as NVRM: Xid (PCI:0000:01:00): 79, ... names the GPU by its PCI address and gives the Xid number. These are the ones that matter for a health check, with the description and first action from NVIDIA's catalogue:

NVIDIA writes the catalogue for Ampere and newer GPUs. On a rented VM you cannot restart the host, and nobody has tested a GPU reset from inside a QuantaCloud VM, so any Xid that calls for a reset, a restart or support is a failed check.

Load every GPU with gpu-burn#

The quickest load test is gpu-burn, which runs the same large matrix multiplication again and again on every GPU and compares each result with the first. A single value that differs by more than 0.001 marks the GPU FAULTY. The project has no releases, so the version is its latest commit, 0d19c94 of 2026-10-02. On a QuantaCloud instance, Docker is already installed, so the Docker build needs no CUDA toolkit on the host. Match CUDA_VERSION to the CUDA version in the nvidia-smi header, labelled CUDA Version up to driver branch 595 and CUDA UMD Version from 610: 13.0.0 for 13.0 or newer, 12.8.1 for 12.8 or 12.9. On an older driver, the Dockerfile's default of 11.8.0 covers every GPU before Blackwell.

git clone https://github.com/wilicc/gpu-burn
cd gpu-burn
docker build --build-arg CUDA_VERSION=13.0.0 -t gpu-burn .
docker run --rm --gpus all gpu-burn ./gpu_burn 600

The last number is the duration in seconds and has to come after any options. The project's own default is 10 seconds, too short to show heat. The options worth knowing:

Without Docker, build it on the host with make COMPUTE=86, using your GPU's compute capability from nvidia-smi --query-gpu=compute_cap --format=csv. That needs nvcc from a CUDA toolkit no newer than the driver's CUDA Version, as checking the driver and CUDA version explains. If the container cannot see the GPUs, running Docker with a GPU covers the NVIDIA Container Toolkit.

While it runs, gpu-burn prints one line that updates in place, with the multiplications done, the Gflop/s, the error count and the temperature of each GPU, and it ends with a verdict per GPU. In the program's own format, with N, X and T standing for the numbers:

100.0%  proc'd: N (X Gflop/s)   errors: 0   temps: T C
Tested 1 GPUs:
	GPU 0: OK

Errors must stay at 0, and (DIED!) after a count means the worker for that GPU crashed. A steady Gflop/s figure through the run matters as much as the verdict: a figure that falls while the temperature climbs points to throttling.

Run NVIDIA DCGM diagnostics#

NVIDIA's Data Center GPU Manager runs a structured set of tests and reports each one as pass or fail. NVIDIA's apt repository has carried version 4.8.0 for Ubuntu 22.04 and 24.04 since 2026-11-07, the newest tag on GitHub is v4.7.0 of 2026-10-21, and the documentation describes 4.7.1. With NVIDIA's repository added (the keyring step in installing NVIDIA drivers on Ubuntu), set CUDA_VERSION to the major number of that same CUDA version in nvidia-smi:

CUDA_VERSION=13
sudo apt-get update
sudo apt-get install --yes --install-recommends datacenter-gpu-manager-4-cuda${CUDA_VERSION}
sudo systemctl enable --now nvidia-dcgm
dcgmi discovery -l
dcgmi diag -r 2

None of the DCGM packages depends on a driver package, so this leaves the installed driver alone. A later sudo apt upgrade can still change the driver if it came from NVIDIA's repository too, so on a running job's machine install only what you name. dcgmi discovery -l lists the GPUs the host engine found, and -r picks the run level. Each level includes the ones below it:

Each test prints a row that reads Pass or Fail, with a line naming the GPU and the condition under a failure. The command exits with 0 when no error was reported, 226 when a test reported an error and 205 when the result calls for isolating the GPU, so a script can act on the exit code.

Two limits apply. NVIDIA documents level 1 only for GPUs outside its data center line unless support is stated for the product. DCGM's packaged test list has entries for the RTX A6000, A100, L40, L40S, H100 PCIe, H200 and RTX PRO 6000 Server Edition, with the H100 PCIe and the RTX PRO 6000 listed under board names, but none for the RTX 6000 Ada, so on that card I rely on gpu-burn for the load. And NVIDIA supports DCGM on bare metal and in VMs with full GPU passthrough, with at least 16 GB of host RAM and as many CPU cores as GPUs. Nobody has run it on a QuantaCloud VM yet, so check that dcgmi discovery -l lists your GPUs before you plan around it.

Watch temperature and clocks under load#

Throttling shows up only under load, so watch it from a second SSH session while gpu-burn runs:

nvidia-smi --query-gpu=index,temperature.gpu,clocks.sm,power.draw,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.sw_thermal_slowdown,clocks_event_reasons.sw_power_cap --format=csv -l 5
nvidia-smi -q -d TEMPERATURE,PERFORMANCE

The second command prints the shutdown, slowdown and maximum operating temperatures for the card, which is what the current temperature should stay below. Passive cards such as the L40S and the RTX PRO 6000 Server Edition depend on the server's airflow, which on a rented VM you cannot change.

Check the topology on multi-GPU VMs#

A multi-GPU VM also needs its links checked, because a training job's speed depends on them:

nvidia-smi topo -m
nvidia-smi topo -p2p r

The first prints the link between every pair of GPUs: NV# for NVLink, and PIX, PXB, PHB, NODE or SYS for paths over PCIe. The second shows whether each pair can read the other's memory directly. Compare both with what the offer said, and read the SXM, PCIe and NVLink guide for what each path means for training.

A short PyTorch test#

The last check uses the software the job will use. This script writes two bit patterns over 80% of each GPU's free memory and reads them back, then repeats one large matrix product for a minute and compares every result with the first. cuBLAS produces bit-identical results on every run on the same GPU while only one stream is active, as here, so any difference is a fault:

import sys, time, torch

seconds = int(sys.argv[1]) if len(sys.argv) > 1 else 60
if not torch.cuda.is_available():
    sys.exit("PyTorch cannot see a GPU: check the driver and the PyTorch build")

failed = False
for i in range(torch.cuda.device_count()):
    dev = torch.device("cuda", i)
    p = torch.cuda.get_device_properties(i)
    free, total = torch.cuda.mem_get_info(i)
    print(f"cuda:{i} {p.name} GPU-{p.uuid}, {total / 2**30:.1f} GiB, compute capability {p.major}.{p.minor}")

    # Write two bit patterns over 80% of the free memory and read them back.
    blocks = [torch.empty(2**30, dtype=torch.uint8, device=dev)
              for _ in range(int(free * 0.8) // 2**30)]
    memory_ok = True
    for pattern in (0x55, 0xAA):
        for block in blocks:
            block.fill_(pattern)
        for block in blocks:
            memory_ok &= int(block.min()) == pattern == int(block.max())
    del blocks
    torch.cuda.empty_cache()

    # Repeat one matrix product and compare every result with the first.
    a = torch.randn(8192, 8192, device=dev)
    b = torch.randn(8192, 8192, device=dev)
    reference = a @ b
    runs = mismatches = 0
    stop = time.time() + seconds
    while time.time() < stop:
        mismatches += int(not torch.equal(a @ b, reference))
        runs += 1
    ok = memory_ok and mismatches == 0 and bool(torch.isfinite(reference).all())
    failed |= not ok
    print(f"  memory patterns {'ok' if memory_ok else 'MISMATCH'}, {runs} products, "
          f"{mismatches} mismatches: {'PASS' if ok else 'FAIL'}")
    del a, b, reference
    torch.cuda.empty_cache()

sys.exit(1 if failed else 0)

Save it as gpu_check.py and run python gpu_check.py 60 in the environment the job will use, after installing PyTorch as the CUDA version guide shows. It prints one line per GPU with its UUID, ends with PASS or FAIL for each, and exits with 1 if any GPU failed. A FAIL on the memory patterns or a single mismatch is a hardware problem, while an error before any line prints is a driver or PyTorch problem.

When a rented GPU fails a check#

The billing rules limit what a failed check costs you to the time the instance ran. Stop it, and the unused seconds of the current hour come back to your balance. If the instance never reached running, or provisioning failed, the whole first hour comes back. Stopping deletes the disk, so copy off anything you need first, including the outputs of the checks.

Then tell support: email support@quantacloud.net or use the ticket form in the console. Include the deployment ID, the GPU's UUID from nvidia-smi -L, the Xid lines and the output of the check that failed. NVIDIA's sudo nvidia-bug-report.sh, which comes with the driver, collects its logs and command outputs if support asks for more. The pricing page and the billing docs have the full billing rules.

My rule: counters and the kernel log on every new instance, gpu-burn for ten minutes before any job longer than a day, DCGM where the GPU supports it, and the PyTorch test in the job's own environment. A GPU that fails any of them is not worth arguing with: stop it, report it, and pick another from the catalog.