InfiniBand and Ethernet now ship at the same port speeds, 400 and 800 Gb/s, so the choice between them is about behaviour under congestion, not raw speed. InfiniBand is built not to drop packets and is managed from one central point. Ethernet carries the same traffic with RoCE plus flow control and tuning, or with the newer Ultra Ethernet transport, which recovers from loss instead of preventing it. My rule: take InfiniBand when one training job spans the whole cluster and nobody on your team has tuned a lossless Ethernet fabric, and take Ethernet when the cluster has to live inside a network you already run, or when it comes as a validated stack such as NVIDIA's Spectrum-X.
This page is about the fabric between servers. The links between GPUs inside one server are a separate subject, covered in InfiniBand vs NVLink.
InfiniBand, RoCE and Ultra Ethernet at a glance#
Three standards carry RDMA between GPU servers today, and two of them run on Ethernet.
| InfiniBand | Ethernet with RoCE | Ultra Ethernet | |
|---|---|---|---|
| Specified by | The InfiniBand Trade Association | The same association for RoCE, first specified in 2010, on top of IEEE Ethernet | The Ultra Ethernet Consortium, a Joint Development Foundation project under the Linux Foundation. Version 1.0 in June 2025, version 1.1 in October 2026 |
| How loss is handled | Link-level flow control, so that packets "are not dropped in the fabric except in the case of serious errors" | Priority Flow Control pauses a traffic class hop by hop, and ECN tells senders to slow down | The transport recovers loss itself, so it runs on best-effort networks as well as lossless ones |
| Spreading load across paths | Adaptive routing in the switches | Flow hashing (ECMP) by default. Adaptive routing on NVIDIA Spectrum-X | Packet spraying across all paths |
| Management | One central fabric manager | Each switch runs its own control plane, as in any Ethernet network | Existing Ethernet and IP routing protocols |
| Port speeds today | 400 Gb/s (NDR) and 800 Gb/s (XDR) | 400 and 800 Gb/s | The Ethernet speeds under it |
| Products | NVIDIA Quantum switches and ConnectX adapters | Switches and adapters from many vendors, NVIDIA's Spectrum-X among them | Timelines that "vary by manufacturer", in the consortium's words |
What RDMA is, and what RoCE adds#
RDMA, remote direct memory access, lets a network card read from and write to an application's memory directly, so data moves between servers "with little CPU involvement", in the InfiniBand Trade Association's words. Distributed training leans on it: NCCL, NVIDIA's library for moving data between GPUs, has one transport for InfiniBand and RoCE, and falls back to IP sockets when that transport is turned off. RDMA is native to InfiniBand.
RoCE, RDMA over Converged Ethernet, carries the same RDMA transport over an Ethernet network. The InfiniBand Trade Association standardizes it and published the first specification in 2010. RoCEv2 is the routable version: it wraps the transport in UDP packets, on port 4791, which the association registered in 2014, so traffic can cross IP subnets. From the application's side the two look alike: the association says RoCE and native InfiniBand "present the same API", and NCCL uses the same transport for both, with NCCL_IB_GID_INDEX selecting the address it uses in RoCE mode.
Why loss and tail latency decide training speed#
Training traffic punishes a network in a way web traffic does not: a step is only as fast as its slowest transfer. Every GPU in the job exchanges gradients (data parallelism) or activations (pipeline parallelism) on each step, and the step ends when the last message lands. The Ultra Ethernet Consortium puts it as "inter-dependency on the progress of individual flows (where tail latency is the figure of merit)".
The traffic is also awkward to balance. Meta, which said in August 2024 that it runs RoCEv2 for most of its AI capacity, describes training traffic as low in entropy (few flows, repeated in the same pattern), bursty at the scale of milliseconds, and made of elephant flows that can each fill a port. Flow hashing (ECMP) performed poorly because there were so few flows to spread. When Meta pinned flows to fixed paths instead, uneven traffic on some uplinks "degraded the training performance up to more than 30%".
Loss makes it worse. The InfiniBand Trade Association's own comparison, written in 2012, says RDMA, "whether native InfiniBand or RoCE, works best when the underlying wires implement a so-called lossless fabric". So the questions to ask of a fabric are what happens when two large flows meet on one link, and who has to do the tuning.
How each fabric handles congestion#
InfiniBand
InfiniBand avoids loss at the link. Its link-level flow control holds a packet back until the next hop can take it, so congestion slows a sender down instead of dropping its packets. A central fabric manager sees the whole fabric and sets up its routing, which is "very much in contrast to traditional Ethernet switched fabrics, which are generally managed autonomously", as the association describes it. NVIDIA's Quantum switches add adaptive routing and congestion control on top, and its SHARP feature moves the reduction step of a collective operation into the switches.
Ethernet with RoCE
Ethernet drops packets when a queue fills, so RoCE needs help. NVIDIA's switch documentation says "RoCE relies on ECN and PFC to operate". Priority Flow Control pauses one traffic class on a link when the receiver's buffer fills. ECN marks packets on a congested switch, and the receiver sends a congestion notification back to the sender, which slows down. NVIDIA's Cumulus Linux switch software sets up both with one command, nv set qos roce, and the adapters in the servers have to support RoCE as well.
It works at scale, with effort. Meta's published account shows how much: it went through path pinning, enhanced ECMP and queue pair scaling to spread flows, turned DCQCN off for its 400 Gb/s clusters and ran on PFC alone, and co-designed its collective library with the network. It also notes "RoCE's more coarse-grained flow control" compared with InfiniBand.
Spectrum-X
Spectrum-X is NVIDIA's Ethernet for AI clusters: Spectrum-X switches and SuperNICs designed and validated as one platform. It is standards-based Ethernet running RoCE, with adaptive routing done by the switch and the adapter together, and congestion control driven by telemetry. NVIDIA's claim for it is 1.6 times the AI network performance of off-the-shelf Ethernet. That is NVIDIA's own figure, and the comparison is with other Ethernet, not with InfiniBand.
Ultra Ethernet
Ultra Ethernet's central piece is a new transport, UET, rather than more tuning of the network under it. UET recovers lost packets itself and sprays the packets of one flow across every available path, which the specification says gives "improved network utilization and reduced tail latency". The specification is blunt about why: "many large-scale lossless fabrics are challenging to operate without triggering head-of-line blocking and congestion spreading". It also defines link-level retry, optional credit-based flow control on links, and optional packet trimming, where a congested switch truncates a packet and forwards what is left as a congestion signal instead of dropping it silently.
Port speeds available today#
Port for port, the two are level. NVIDIA's ConnectX-8 adapter runs either protocol at up to 800 Gb/s, and the DGX B300 ships with eight of them, one per GPU.
| 400 Gb/s | 800 Gb/s | Next | |
|---|---|---|---|
| InfiniBand | NDR. NVIDIA Quantum-2 switches with 64 ports, ConnectX-7 adapters | XDR. NVIDIA Quantum-X800 switches with 144 ports, ConnectX-8 adapters | GDR at 1.6 Tb/s, on the InfiniBand Trade Association's roadmap for the 2028 timeframe |
| Ethernet | 400 GbE. NVIDIA's B300 design on Spectrum-4 gives each GPU 2 x 400 GbE | 800 GbE, an IEEE standard since 16 February 2024. NVIDIA's SN6600 has 128 ports of 800G | 1.6 Tb/s. The IEEE project, P802.3dj, was still an active task force on 2026-11-12. NVIDIA lists the ConnectX-9 at 1.6 Tb/s per GPU for Vera Rubin systems |
Whether you can get 800 Gb/s is a separate question from whether it exists. For its September 2026 ratings, SemiAnalysis asked every provider it tested for "800G RoCE or XDR InfiniBand", and noted that "many providers did not have it yet".
Where the two ecosystems stand#
NVIDIA sells both and publishes its own reference cluster on both. Its DGX SuperPOD designs for the H100 and B200 use an InfiniBand compute fabric. For the B300 there are two designs: one on Quantum-X800 InfiniBand, and one on Spectrum-X Ethernet with two 400 GbE ports per GPU on separate planes.
Outside NVIDIA's designs, large Ethernet training fabrics are on the public record too. Meta chose RoCEv2 "for the majority of our AI capacity", in clusters of thousands of GPUs each. In SemiAnalysis's 2026 testing, every provider but one used InfiniBand or RoCE, and the exception was a hyperscaler with a fabric of its own.
Ultra Ethernet is a published specification, and it is moving quickly.
| Version | Released | What it is |
|---|---|---|
| 1.0 | 11 June 2025 | The first public release |
| 1.0.1 | 5 September 2025 | Editorial clarifications and one algorithm correction |
| 1.0.2 | January 2026 | Corrections to the congestion control algorithms |
| 1.0.3 | 16 July 2026 | 200 Gb/s per lane signaling, and corrections to the link layer |
| 1.1 | 12 October 2026 | In-network collectives, a profile for scale-up networks, programmable congestion management, congestion signaling and a smaller forwarding header |
Version 1.1 keeps "full backward compatibility with previous versions". A specification is not a product, though. The consortium's own answer on availability is that "Product availability and implementation timelines vary by manufacturer", so ask a vendor which version its switches and adapters implement, and which of the optional features.
What the published comparisons say#
I do not have an independent test of InfiniBand against RoCE on the same GPUs and the same job that I would quote as a number. What is published comes from three kinds of source, and none of them is a neutral head-to-head.
| Source | What it compared | What it found |
|---|---|---|
| NVIDIA, on its Spectrum-X page | Spectrum-X against off-the-shelf Ethernet | 1.6 times the AI network performance. The vendor's own figure |
| Meta, August 2024 | Its own RoCE fabric, before and after tuning | Enhanced ECMP with queue pair scaling improved the AllReduce collective by up to 40% over baseline ECMP |
| SemiAnalysis, October 2024 | InfiniBand, Spectrum-X and Ethernet from other switch vendors, as an analyst's judgment | NVIDIA's two options cost more. The belief that Ethernet performs far worse "does not reflect reality", but closing the gap takes NCCL tuning that is not easy "unless you have good engineering talent and time" |
My reading of the three: an untuned Ethernet fabric is slower, a tuned one closes most of the gap, and the tuning is the hard part. That is why I would settle it with a test, not an argument. Ask for an all_reduce_perf run from NVIDIA's nccl-tests across every node before handover, on whichever fabric you choose.
When Ethernet is enough and when InfiniBand earns its cost#
Here is the rule of thumb I use, by situation.
| Your situation | What I would pick | Why |
|---|---|---|
| Inference, or training that fits in one 8-GPU server | No cluster fabric: one dedicated server | The traffic stays inside the server. See InfiniBand vs NVLink |
| Multi-node training, a small team, nobody who has run PFC and ECN in production | InfiniBand | It does not drop packets under congestion by design, it is managed from one place, and it is the compute fabric in NVIDIA's reference designs for the H100 and B200, and one of the two for the B300 |
| A cluster that must join an Ethernet network your team already runs and monitors | Ethernet with RoCE, as a validated stack such as Spectrum-X | The same tools and staff, and NVIDIA publishes a B300 design on it |
| A large cluster built by a team with its own network engineers | Ethernet | Meta's fabric shows it works at that scale, and NVIDIA says a multiplane Spectrum-X network reaches 128,000 GPUs in two tiers |
| The front-end network, for internet access, the scheduler and storage traffic | Ethernet | The front end is, in SemiAnalysis's words, "simply a normal ethernet network" |
The cost InfiniBand has to earn is real but not one I can put a number on. SemiAnalysis wrote in 2024 that NVIDIA's fabrics cost more than Ethernet from other vendors, and that on InfiniBand, "to have a decent REST interface, you need to buy UFM management licenses". It still recommended InfiniBand or Spectrum-X to the providers it was writing for. Switches, optics and licences are quoted per build, so compare two written quotes for the same node count, not list prices.
What to put in a cluster brief#
QuantaCloud builds clusters to order with either fabric: InfiniBand or RoCE Ethernet between servers, with the configuration, lead time and terms in writing before you commit. On-demand instances are single VMs with up to 8 GPUs, so the fabric question only comes up for a reserved build.
Name four things in the brief: the fabric, the port speed per GPU (400 or 800 Gb/s), whether storage and management get their own networks, and the acceptance test. GPU clusters covers node platforms and sizing, and Slurm vs Kubernetes covers the scheduler that will hand the fabric to your jobs.
Send a cluster briefInfiniBand and Ethernet questions#
What is RoCE?
RoCE is RDMA over Converged Ethernet: the direct memory transfers that InfiniBand does, carried over an Ethernet network. The InfiniBand Trade Association standardizes it and published the first specification in 2010.
What is RoCEv2?
RoCEv2 is the routable version of RoCE. It carries the InfiniBand transport inside UDP packets on port 4791, so traffic can cross IP subnets, and it is the version Meta's training clusters and NVIDIA's switch documentation describe.
Is InfiniBand faster than Ethernet?
Not per port. Both run at 400 and 800 Gb/s today, and NVIDIA's ConnectX-8 adapter does either at up to 800 Gb/s. The difference is in how each handles congestion and how much tuning it needs to keep the slowest flow fast.
What is Ultra Ethernet?
Ultra Ethernet is an open specification from the Ultra Ethernet Consortium for an Ethernet transport built for AI and high-performance computing. Its transport recovers loss itself and sprays packets across all paths, so it does not depend on a lossless network. Version 1.1 was published on 12 October 2026.
What is Spectrum-X?
Spectrum-X is NVIDIA's Ethernet platform for AI clusters: Spectrum switches and ConnectX SuperNICs validated together, running RoCE with adaptive routing and telemetry-based congestion control. NVIDIA puts its AI network performance at 1.6 times that of off-the-shelf Ethernet.
Does NCCL work over Ethernet?
Yes. NCCL runs over RoCE through the same code path it uses for InfiniBand, and NCCL_IB_GID_INDEX sets the address index it uses in RoCE mode.
Can I rent an InfiniBand cluster by the hour?
Not from QuantaCloud. On-demand instances are single VMs with 1, 2, 4 or 8 GPUs, and a multi-node cluster on InfiniBand or RoCE is a reserved build for a term.
The rule I follow: decide the fabric by who will run it, not by the logo. If one job spans every node and you want the design NVIDIA itself tests, ask for InfiniBand. If the cluster has to fit an Ethernet operation you already have, ask for RoCE on a validated stack and hold it to the same acceptance test. Either way, put the fabric, the port speed and the test in the brief, and the quote comes back in writing.