Slurm is the scheduler for jobs that end, and Kubernetes is the scheduler for services that stay up. For a GPU cluster that mostly trains models, pick Slurm: it hands a job all of its nodes at once, shares a fixed pool fairly between people, and comes with job arrays and backfill. For one that mostly serves models, pick Kubernetes: it keeps replicas running, rolls out new versions and scales with traffic. If the cluster does both, you no longer have to choose, because SchedMD's Slinky project runs Slurm on Kubernetes.
SchedMD, which develops Slurm, puts the split better than I can. Kubernetes "excels at scheduling workloads that run for an indefinite amount of time, with potentially vague resource requirements, on a single node, with loose policy", and Slurm "excels at quickly scheduling workloads that run for a finite amount of time, with well-defined resource requirements and topology, on multiple nodes, with strict policy, and a known resource pool".
Which scheduler for which workload#
Decide by what the GPUs will spend most of their hours doing.
| Workload | Pick | Why |
|---|---|---|
| Training runs that span several servers | Slurm | A job is granted its whole allocation at once and keeps it until it ends or hits its time limit |
| Many people sharing a fixed pool of GPUs | Slurm | Fair-share priority, accounts, limits and backfill are part of the scheduler |
| Sweeps and batch evaluation | Slurm | One job array submits thousands of similar jobs |
| Inference endpoints that stay up and follow traffic | Kubernetes | Deployments keep replicas running, and the Horizontal Pod Autoscaler adds and removes them |
| Many small services around the models, such as gateways, queues and web apps | Kubernetes | Deployments, Services, health probes and rolling updates are built in |
| Training and serving on the same hardware | Both | Slurm on Kubernetes through Slinky, or two pools of nodes |
If every job fits on one 8-GPU server, you may need neither: one dedicated server and a launcher such as torchrun do the job.
What Slurm does well#
Slurm's model is the allocation: you ask for nodes, GPUs and a time limit, and the job waits in a queue until all of it is free. Its documentation describes "an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters", and NVIDIA says more than half of the top 10 and top 100 systems on the TOP500 list use it.
# 16 GPUs across two 8-GPU nodes, for at most 24 hours
sbatch --nodes=2 --gpus-per-node=8 --time=24:00:00 train.sh
# a 100-job sweep, one GPU each
sbatch --array=0-99 --gpus=1 sweep.sh
Inside the allocation, train.sh runs the same training launch you would test on one server first, such as torchrun with DDP or FSDP or DeepSpeed, spread across the nodes it was given.
Four built-in features matter most on a shared GPU cluster. Fair share comes from the default priority plugin, which can weigh how much each account has already used, and has used the Fair Tree algorithm for it since Slurm 19.05. Give that factor a weight in slurm.conf and a team that has used less than its share moves up the queue. Backfill starts small jobs in the gaps left while a large one waits. Job arrays submit collections of similar jobs in one command, and the documentation says arrays "with millions of tasks can be submitted in milliseconds". Topology-aware selection places a job on nodes that sit close together on the network, which matters once the fabric has more than one switch tier.
The current stable release is Slurm 26.05.6, published on 2026-11-10, with 26.11 in release candidate. SchedMD has released a major version every six months since 24.05. NVIDIA acquired SchedMD in December 2025 and said it "will continue to develop and distribute Slurm as open-source, vendor-neutral software".
What Kubernetes does well#
Kubernetes keeps a declared state true: this many replicas of this container, behind this address, restarted when they fail. That is the shape of an inference service, such as a vLLM container serving an API. A HorizontalPodAutoscaler "automatically updates a workload resource", in the documentation's words, "with the aim of automatically scaling capacity to match demand", and it comes with a large ecosystem of operators and monitoring tools. Slurm's own documentation concedes the point: Kubernetes is being adopted in HPC clusters "to orchestrate deployments" and "run certain workloads (e.g. AI/ML inference)".
What Kubernetes does not do by default is batch. Its scheduler "looks for Pods not yet bound to a node, and assigns each Pod to a suitable node", and by default it has no fair-share queue between teams and no notion of a job that needs sixteen GPUs or none. The current release is v1.37.2, published on 2026-10-28, and the project maintains its three most recent minor releases with about a year of patches each, so plan on an upgrade at least once a year.
How each one handles GPUs#
Both schedule NVIDIA GPUs as countable devices. They differ in who sets up the node.
| Slurm | Kubernetes | |
|---|---|---|
| How a job asks for GPUs | --gpus, --gpus-per-node, --gpus-per-task or --gres | nvidia.com/gpu under limits, or a resource claim with Dynamic Resource Allocation |
| How the scheduler learns what a node has | GRES entries in slurm.conf and gres.conf, where AutoDetect=nvml finds NVIDIA GPUs | NVIDIA's device plugin, through the device plugin framework, stable since v1.26. Or the DRA driver for NVIDIA GPUs, v0.5.1, through Dynamic Resource Allocation, stable since v1.35 |
| Who installs drivers and the container toolkit | You, on every node | The NVIDIA GPU Operator, v26.7.2, which automates the driver, the container toolkit, the device plugin, node labels and DCGM monitoring |
| Multi-node jobs that start all at once | Built in: it is how an allocation works | An add-on such as Kueue, or Kubernetes' own gang scheduling, beta since v1.37 and off by default |
| Fair sharing between teams | Built in, through the multifactor priority plugin | An add-on, such as Kueue or KAI Scheduler |
| Splitting one GPU | MPS, MIG and sharding, each covered in Slurm's GRES guide | Time-slicing and MIG, configured through the GPU Operator |
Watch one naming trap: gang scheduling means two different things. In Slurm's documentation it is time slicing, where "two or more jobs are allocated to the same resources" and take turns. In Kubernetes it means an all-or-nothing start: the documentation says gang scheduling "ensures that a group of Pods are scheduled on an 'all-or-nothing' basis". When someone says Kubernetes lacks gang scheduling, they mean the second, which Slurm does for every allocation.
The batch schedulers that sit on Kubernetes#
Four open-source projects add the queue that Kubernetes lacks, and each has shipped a release in the last six weeks.
| Project | Home | Latest release on 2026-11-12 | What it adds |
|---|---|---|---|
| Kueue | A Kubernetes special interest group project | v0.20.5, 2026-11-05 | Queues, quota, fair sharing and all-or-nothing admission. It "works alongside kube-scheduler" instead of replacing it |
| Volcano | Cloud Native Computing Foundation, incubating | v1.16.0, 2026-10-14 | A batch scheduler that extends the standard one |
| KAI Scheduler | Cloud Native Computing Foundation, sandbox. Its repository used to sit under NVIDIA's GitHub organization | v0.19.0, 2026-11-11 | A scheduler built around GPU allocation and fairness, which can run beside other schedulers |
| Apache YuniKorn | Apache Software Foundation | v1.10.0, 2026-10-06 | A general resource scheduler that runs as a custom Kubernetes scheduler |
A scheduler needs a job format to schedule. For training that is Kubeflow Trainer (v2.4.0) or JobSet (v0.13.0), and for a model too large for one server, served from pods that must start together, it is LeaderWorkerSet, which reached v1.0.0 on 2026-10-16. Multi-GPU vLLM explains how such a model is split inside and across nodes. That is three layers to choose and keep compatible before the first training job runs: Kubernetes, a batch scheduler and a job API.
Slurm on Kubernetes with Slinky#
Slinky is SchedMD's answer to running both, and it comes in two parts. The slurm-operator runs a Slurm cluster as Kubernetes pods: the controller, sets of worker nodes and login nodes. Its latest stable tag is v1.2.4, on a commit dated 2026-10-28, and it supports Kubernetes 1.35 to 1.37. Worker sets can scale to zero and pair with a Horizontal Pod Autoscaler, and the operator supports hybrid clusters where some Slurm components run outside Kubernetes.
The slurm-bridge goes the other way: it makes Slurm a scheduler for Kubernetes, so pods and Slurm jobs share one pool of nodes under one set of policies. Its documentation is clear about the limit: on shared nodes "the hardware is shared over time", so a node runs Slurm jobs or bridged Kubernetes workloads, not both at once.
Slinky is already in use at GPU clouds: two of the provider reviews in SemiAnalysis's September 2026 ClusterMAX report describe clusters tested through Kubernetes and Slinky.
What each one costs to run#
Neither is free to operate, and the work is of a different kind.
| Slurm | Kubernetes | |
|---|---|---|
| Daemons | slurmctld, with an optional backup, slurmd on every node, and slurmdbd with MySQL or MariaDB if you want accounting | An API server, etcd, a scheduler and a controller manager, then a kubelet and a container runtime on every node |
| What must stay in step | Clocks, users and groups across every node, and one authentication key | Certificates, API versions and the versions of every add-on |
| The GPU stack | Driver and CUDA installed on each node, plus the GRES configuration | The GPU Operator, plus whichever batch scheduler and job API you chose |
| Upgrades | A major release every six months. In-place upgrades work from the previous three | Three minor releases maintained at a time, each with about a year of patches |
| What users need to learn | SSH to a login node, sbatch and squeue, and a shared file system | Containers, manifests and kubectl |
The honest summary is that Slurm is less software and more convention, and Kubernetes is more software and more automation. A research group whose members already write sbatch scripts has nothing new to learn on Slurm. A platform team that already runs Kubernetes should not stand up a separate Slurm cluster beside it: add Kueue, or run Slurm inside Kubernetes with Slinky.
Hybrid patterns for running both#
These are the four patterns I would choose between.
| Pattern | How it works | When it fits |
|---|---|---|
| Two pools | Some nodes run Slurm and the rest run Kubernetes | Training and serving loads are both steady, and you can live with moving nodes by hand |
| Slurm inside Kubernetes | The slurm-operator runs Slurm as pods on a Kubernetes cluster | A platform team runs Kubernetes, and researchers want sbatch |
| One pool, Slurm places everything | The slurm-bridge schedules pods and Slurm jobs onto the same nodes | You want one set of fair-share and priority rules for both kinds of work |
| Kubernetes only | Kueue, Volcano or KAI Scheduler supplies the queue | Serving is the main job and training is occasional |
What to specify when you order a cluster#
The scheduler belongs in the brief, because it changes the build. QuantaCloud builds GPU clusters to order: we order and build the hardware to your spec, and the configuration, lead time and terms come back in writing. On-demand instances are single VMs, and QuantaCloud offers no managed Kubernetes, so a multi-node scheduler only comes up for a reserved build. Reserved capacity walks through how a build is quoted.
| Put in the brief | Why it changes the build |
|---|---|
| Which scheduler: Slurm, Kubernetes or both | It decides the control and login nodes the cluster needs beside the GPU nodes |
| Who will install and operate it | Ask for this in writing with the quote |
| Shared storage for home directories and datasets | Slurm users expect the same paths on every node, and Kubernetes needs a storage class for shared volumes |
| The fabric and how jobs reach it | InfiniBand or Ethernet, and RDMA devices exposed to jobs or pods |
| The operating system, driver and CUDA versions | They are part of the build, and every GPU node should match |
| An acceptance test through the scheduler | A multi-node all_reduce_perf run submitted as a job, not started by hand |
While the cluster is built, you can get the training script right on one on-demand VM from the GPU catalog, with up to 8 GPUs when the catalog lists them. The deploy guide covers launching one.
Send a cluster briefSlurm and Kubernetes questions#
Is Slurm or Kubernetes better for AI training?
Slurm, for training that spans several nodes on a fixed cluster. It starts a job only when the whole allocation is free, and fair sharing and job arrays come with it. Kubernetes can do the same with a batch scheduler such as Kueue on top.
Can Slurm run on Kubernetes?
Yes. SchedMD's slurm-operator, part of the Slinky project, runs a Slurm cluster as Kubernetes pods, and its latest stable version on 2026-11-12 was v1.2.4. The companion slurm-bridge lets Slurm schedule Kubernetes pods.
Does Kubernetes support GPUs?
Yes. GPU scheduling through device plugins has been stable since Kubernetes v1.26: a pod asks for nvidia.com/gpu under limits. The NVIDIA GPU Operator installs the driver, the container toolkit and the device plugin on each node.
What is a Slurm cluster?
A Slurm cluster is a set of Linux servers managed by one Slurm controller. Each server runs the slurmd daemon, users submit jobs from a login node, and the controller queues the jobs and allocates nodes and GPUs to them.
Who owns Slurm?
SchedMD is Slurm's lead developer, and NVIDIA acquired SchedMD in December 2025. Slurm remains open source, and NVIDIA said it will keep distributing it as vendor-neutral software.
My rule: count the hours. If most GPU-hours go to jobs that start, run and end, build the cluster around Slurm and add Kubernetes only if you serve models from it. If most go to services that stay up, build it around Kubernetes and add a batch queue. Name the scheduler in the brief either way, together with the node platform and the fabric, and ask who runs it.