The InfiniBand vs Ethernet decision for an AI cluster is settled by three things that are not on any specification sheet: how many GPU nodes you are connecting, whether the workload is collective-heavy training or inference and serving, and whether anyone on your team has run a lossless Ethernet fabric before. At the scale of one or two racks, tuned RoCE Ethernet is almost always the right answer. At several hundred nodes running large-scale training, InfiniBand still wins on the behaviours that are hardest to configure your way into. The interesting decisions are all in the middle, and that is what this guide is about.

It is written for infrastructure engineers and IT buyers who have already chosen their GPU servers — if you have not, start with choosing a GPU server for an AI workload — and now have to specify the network between them. By the end you will know which fabric your cluster size and workload actually argue for, why published latency figures for the same hardware differ by more than twenty times, how to size leaf and spine from the rail count, which optics will physically refuse to seat in which cage, and the single management decision that determines whether an InfiniBand fabric comes up at all.

Every specification below is cited to a manufacturer document, and where the manufacturer does not publish a number, this guide says so rather than borrowing one from somebody who does not know either.

InfiniBand vs Ethernet: the short answer

Choose InfiniBand when the cluster is large, the workload is training, and collective operations dominate the job's wall-clock time. Choose Ethernet when the cluster is small to mid-sized, the workload is inference or mixed, the same fabric has to carry storage and general traffic, or your team already operates Ethernet at scale.

The reason is not raw speed. Both fabrics carry 400 and 800 gigabits per second per port in current hardware, and both do RDMA. The difference is where the work sits. InfiniBand arrives with a lossless link layer, a fabric-wide manager, adaptive routing and congestion control as defaults, plus in-network collective offload. Ethernet reaches comparable behaviour through RDMA over Converged Ethernet — RoCEv2 — which NVIDIA's own switch operating system documentation describes plainly: "RoCE relies on explicit congestion notification (ECN) and priority flow control (PFC) to operate." Those are settings. Someone has to choose them, apply them consistently across switch, NIC, driver and host, and prove they hold under congestion.

So the honest framing of InfiniBand vs Ethernet for AI is not "which is faster." It is how much fabric engineering you want to own. InfiniBand sells you a fabric. Ethernet sells you the parts of one.

How the InfiniBand vs Ethernet decision is actually made

Four variables decide it in practice: node count, workload shape, existing skills, and what else the network has to carry. Everything else is detail hanging off those four.

Node count sets the shape. Below about 32 nodes a single leaf layer covers the cluster and there is no topology to manage, so InfiniBand's fabric-management advantages have little to work on. Above a few hundred nodes, path count, failure modes and the sensitivity of collective operations to one congested link all grow faster than the node count — the regime InfiniBand was built for.

Workload shape sets the stakes. Distributed training synchronises gradients constantly, and an all-reduce that stalls stalls every GPU in the job; inference and serving are latency-sensitive per request but not collectively coupled. Existing skills are the variable buyers discount most: running Ethernet is not the same discipline as running lossless Ethernet, and PFC, ECN, traffic-class mapping and buffer allocation have to line up across switch, NIC, driver, firmware and application before RDMA behaves. What else the network carries settles the rest — if the same fabric must serve storage, provisioning and general east-west traffic, Ethernet's ubiquity is a real advantage; if the compute fabric is dedicated, as it is in every reference design of any size, that advantage narrows.

Axis	InfiniBand (NVIDIA Quantum)	Ethernet (NVIDIA Spectrum / Dell PowerSwitch)	What it means when you buy
RDMA path	Native; the transport is the fabric	RoCEv2, which "relies on explicit congestion notification (ECN) and priority flow control (PFC) to operate"	On Ethernet the RDMA path is a configuration project you must validate, not a property you purchase
Fabric management	A subnet manager is mandatory — on-board on some switches, from NVIDIA UFM or a host otherwise	Conventional switch management; no fabric-wide controller required	An InfiniBand fabric passes no traffic until a subnet manager has run. See the managed-versus-unmanaged section
In-network compute	SHARP — collective operations offloaded "from CPUs and GPUs to the network"; SHARPv3 on Quantum-2, SHARP v4 on Quantum-X800	No direct equivalent; Spectrum-X instead adds packet-by-packet adaptive routing with endpoint reordering	Only matters if collectives dominate your job time. For inference, it is close to irrelevant
Port speeds in current hardware	400 Gb/s NDR, 800 Gb/s XDR	400GbE and 800GbE	Neither fabric is bandwidth-limited at this tier. Do not decide on line rate
Connector	OSFP throughout the Quantum line	Splits: QSFP-DD, OSFP, QSFP56-DD and OSFP112 across models	Your existing optic inventory may not carry over at all. See the cabling section
Supply	Switch silicon effectively single-source	Multiple silicon vendors and multiple switch vendors	A procurement and lead-time question as much as a technical one
Operating systems	NVIDIA MLNX-OS on Quantum-2; NVOS on Quantum-X800	NVIDIA Cumulus Linux, Enterprise SONiC or Dell SmartFabric OS10 depending on model	Open-networking hardware means the box and the licence are separate purchases — critical on the secondary market

Why InfiniBand vs Ethernet latency figures disagree by more than twenty times

Because three different measurements are being published under one word. You will find InfiniBand latency quoted at under 100 nanoseconds, at one to two microseconds, and Ethernet quoted at ten to fifty microseconds — a spread of more than twenty times on the number that supposedly decides the argument. None of those figures is invented. They are measured at three different layers, and almost nobody says which.

The decisive fact sits in what the manufacturer does not publish. NVIDIA states no port-to-port latency for Quantum-2 anywhere in its own documentation. The QM97xx hardware user manual specification table lists throughput, power, weight, dimensions, operating temperature, acoustics and connector cages — and contains no latency row at all. Neither does the Quantum-2 platform page. That absence is why third-party pages quote the figure four different ways without ever agreeing.

Measurement layer	Figures in circulation	What is actually being measured	Who publishes it
Switch port-to-port (cut-through)	Roughly 90–100 ns for InfiniBand; sub-microsecond for Ethernet	One hop through one switch ASIC, nothing else	Third-party pages for InfiniBand — NVIDIA publishes none. Dell does publish port-to-port figures for its Ethernet switches on its own spec sheets
End-to-end RDMA (application or MPI)	Low single-digit microseconds on both fabrics, InfiniBand ahead	Host adapter, protocol stack, software and the wire, all together — the number an application experiences	Third-party benchmark summaries; no manufacturer publishes a comparable pair
TCP/IP without RDMA	Tens of microseconds	The conventional networking stack — not RoCE, not either fabric's RDMA path	Comparisons that quote this as "Ethernet" are comparing an RDMA fabric to a non-RDMA one

Two rules follow, and they are worth more than any of the numbers. First: a switch-forwarding figure and an application figure are three orders of magnitude apart and are never comparable. Second: if a comparison quotes tens of microseconds for "Ethernet", it has silently swapped RoCE for TCP/IP and the conclusion it draws is about a fabric nobody is proposing to build. Ask which layer, or discard the number.

When RoCE is enough — the RoCE vs InfiniBand question, answered by an operator

RoCE is enough at very large scale — the largest publicly documented AI training network runs on it — but the evidence for that also shows exactly where the effort goes. Meta has published a detailed account of using RoCE for distributed AI training at scale, stating that it uses "RoCEv2 as the inter-node communication transport for the majority of our AI capacity" and training Llama 3 on a RoCE cluster built from standard Ethernet silicon.

Three details from that account are worth more to a buyer than any vendor benchmark. Meta "proceeded without DCQCN for our 400G deployments", running on priority flow control alone and reporting stable performance over more than a year — the textbook congestion-control mechanism turned out not to be the load-bearing part. What was load-bearing was routing: path-pinning produced uneven traffic distribution that degraded training "up to more than 30%", and better load balancing across queue pairs recovered "up to 40% for the AllReduce collective." And bandwidth upgrades were treated as an expensive temporary mitigation rather than a fix.

Read the InfiniBand vs RoCE trade honestly and that is the answer: the fabric type did not decide the outcome — the tuning did. RoCE is viable at a scale far beyond anything most buyers will build. It is a sustained engineering commitment, not a purchase. Budget accordingly, and note that the Ultra Ethernet Consortium's Specification 1.0 exists precisely to move that work out of each operator's hands and into the standard, covering transport, congestion control, RDMA, link and PHY technologies and security across NICs, switches, optics and cables.

Is InfiniBand proprietary, and who owns it?

The specification is open. The supply is not. These two facts are usually collapsed into one, and separating them is the whole of the lock-in question.

InfiniBand is maintained by the InfiniBand Trade Association, whose steering committee includes HPE, IBM, Intel and NVIDIA, and it was the IBTA — not NVIDIA — that published the XDR specification defining 800 Gb/s per port. In that sense InfiniBand is no more proprietary than Ethernet.

What is concentrated is who makes the silicon. NVIDIA acquired Mellanox, the last independent InfiniBand switch supplier, and in practice NVIDIA is now where InfiniBand switches come from. Open standard, single effective source: that is a supply-chain and pricing risk, not a technical one, and it should be assessed as such rather than folded into an argument about the protocol.

The three networks in a GPU pod

Buyers price one network and then discover they have to build three. The compute fabric is the one everybody shops for. The storage network and the management network are the ones that appear halfway through the deployment, and in NVIDIA's own reference design there are in fact four.

The compute fabric carries gradient and collective traffic between nodes. It is the highest-bandwidth, most latency-sensitive network in the pod, and it is the one this guide's fabric decision is about. In a rail-optimised design each GPU gets its own link out of the node, so an eight-GPU server contributes eight compute-fabric ports, not one.

The storage network feeds the training data in, on its own switches. NVIDIA's DGX SuperPOD reference architecture states that per-node storage throughput "must exceed 40 GBps", with node connections deliberately oversubscribed "with a ratio near 4:3" while the storage devices themselves get a 1:1 port-to-uplink ratio — a separate purchase with a separate sizing calculation. If you are still deciding the architecture behind it, our guide to SAN, NAS and DAS storage architectures covers the layer underneath. The in-band management network then carries host OS access, provisioning and telemetry: conventional Ethernet, typically 100GbE in that same reference architecture.

The out-of-band management network is the one that gets forgotten, and it is the cheapest to get wrong. It reaches BMCs, PDUs and console ports and stays up when everything else is down. NVIDIA's reference architecture specifies SN2201 switches for exactly this role, and Dell describes the Dell PowerSwitch SN2201 in its own words as an "Out-of-band (OOB) management switch, connecting up to 48 x 1G Base-T host ports with non-blocking 100 GbE spine uplinks." It is a 1-gigabit access switch with 100GbE uplinks, drawing 98 W typical, and it will never carry AI traffic. It is also mandatory, and a pod budget that omits it is short by one switch per rack.

The practical consequence: when you cost an AI pod, cost four networks. A comparison that weighs an InfiniBand compute fabric against an Ethernet compute fabric and stops there has left out the majority of the switch count. And if you are still selecting the nodes those networks attach to, our guide to Dell PowerEdge GPU servers covers the compute side of the same pod.

Managed or unmanaged: the QM9700, the QM9790 and the subnet manager

An InfiniBand fabric does not self-assemble. A subnet manager has to discover the topology, assign local identifiers and program the forwarding tables before a single packet moves — and the switch model you buy determines whether you have one. The QM9700 and the QM9790 are the same 1U switch in every respect except this, and it is the difference that decides whether the fabric comes up.

This gets published both ways, which is why it is worth settling from the manufacturer's document. You will find the externally managed model listed in vendor comparison tables with a subnet-manager tick in the same column as the managed one — and, on the same vendor's own product page, a customer answer advising buyers who need a built-in subnet manager to choose the managed model instead. Both cannot be true. NVIDIA's hardware manual resolves it, and the resolution is physical rather than a matter of licensing.

Interface or capability	QM9700 (internally managed)	QM9790 (externally managed)
On-board subnet manager	Yes — NVIDIA rates it for bringing up "up to 2,000 nodes", running MLNX-OS	None. Expects NVIDIA UFM, or a host-based subnet manager, elsewhere in the fabric
Management port	Front, 1 port	Not available
Console port	Front	Not available
USB port	Front, USB 3.0 type A	Not available
Management interfaces	CLI, WebUI, SNMP and JSON	Via UFM, from elsewhere
Ports, cages, capacity, chassis	Identical — 64 × 400 Gb/s NDR across 32 OSFP cages, 2 power supplies, 7 fans, same 1U chassis
Typical power, passive cables (ATIS)	747 W	640 W
Maximum power, active cables	1,720 W	1,610 W

That power row is the cleanest proof available that the difference is hardware and not a feature flag: the managed switch draws 107 W more at typical load for exactly the same 64 ports. That is the on-board management complex, measured, in NVIDIA's own specification table.

There are three legitimate ways to get an InfiniBand subnet manager, and this is not a story about one switch being broken:

1. On board an internally managed switch, such as the internally managed NVIDIA Quantum-2 QM9700 — simplest for a single-switch or small fabric.
2. From NVIDIA Unified Fabric Manager, running centrally — the normal choice at scale, where putting a subnet manager in every switch is redundant and, at 107 W a time, not free.
3. From a host running a software subnet manager on a server attached to the fabric.

Plenty of production fabrics run entirely on externally managed switches by design, and that is the correct architecture at scale. The failure mode is narrower and more specific: buying the externally managed QM9790 as your only InfiniBand switch, with no UFM and no host-based manager in the design. The fabric will not come up, and there is no console port on the box to fix it from locally. Decide where your subnet manager lives before you decide which switch to order — not after.

The commercial shape of this is worth knowing too. Against published new-channel list pricing, the internally managed model runs at roughly 1.6× the externally managed one — a premium of about 58 % on an otherwise identical switch. In a fabric of any size that arithmetic usually argues for one managed switch and the rest unmanaged, or for UFM and none at all. In a single-switch deployment it argues for the managed model without much debate.

The InfiniBand lineup: NVIDIA Quantum-2 and Quantum-X800

Two generations are current: Quantum-2 at 400 Gb/s NDR InfiniBand, and Quantum-X800 at 800 Gb/s XDR. Anything describing HDR at 200 Gb/s as the latest InfiniBand is describing the generation before last. Every figure in the table below comes from NVIDIA's QM97xx hardware user manual, its management-interface table and its XDR switch systems user manual.

Specification	QM9700	QM9790	Q3400-RA
Generation and rate	Quantum-2, NDR 400 Gb/s	Quantum-2, NDR 400 Gb/s	Quantum-X800, XDR 800 Gb/s
Ports	64 × 400 Gb/s	64 × 400 Gb/s	144 × 800 Gb/s
Physical cages	32 OSFP	32 OSFP	72 OSFP
Breakout	to 128 × 200 Gb/s	to 128 × 200 Gb/s	not stated by NVIDIA
Throughput as NVIDIA states it	51.2 Tb/s bidirectional, or "Switching: 25.6Tbps" in the hardware manual	Same	115.2 Tb/s, stated unidirectionally
Rack units	1U	1U	4U
Power supplies / fans	2 / 7	2 / 7	8 / 10
Typical power (passive, ATIS)	747 W	640 W	not published by NVIDIA
Input voltage	200–240 Vac	200–240 Vac	not published by NVIDIA
Subnet manager	On board, up to 2,000 nodes	None — expects UFM	NVOS, with a dedicated in-band management port
In-network compute	SHARPv3	SHARPv3	SHARP v4

The Quantum-2 pair carries NVIDIA's claim of more than 66.5 billion packets per second and supports 40, 56, 100, 200 and 400 Gb/s per port. On the XDR side, NVIDIA Quantum-X800 Q3400-RA presents 144 ports of 800 Gb/s over 72 OSFP cages in 4U, with eight power supplies, ten replaceable fans and a separate in-band management port; its smaller sibling the Q3200-RA is 72 ports over 36 cages in 2U — the same cage-doubling rule one tier down. NVIDIA claims SHARP v4 on Quantum-X800 delivers "2x Higher Bandwidth", "5x Data Throughput" and "9x In-Network Computing", but does not state the baseline on the page that makes those claims: treat them as vendor positioning rather than benchmark.

One naming note that catches buyers: the "-RA" suffix on Q3400-RA means air-cooled, not reverse airflow. NVIDIA describes the RA models as an "air-cooled system designed for standard 19-inch rack cabinets." The airflow-direction token on the Quantum-2 line is a different suffix entirely, covered in the secondary-market section.

How to read a switching-capacity figure — including when the vendor gives you two

Divide the stated capacity by (ports × port speed). If you get 2, the figure is bidirectional. If you get 1, it is not. Never compare two capacity numbers without doing that division first, because vendors do not use one convention — and neither does any single vendor.

NVIDIA publishes both figures for the same QM9700 switch. Its Quantum-2 platform page states "51.2 terabits per second (Tb/s) of bidirectional throughput"; its QM97xx hardware user manual lists "Switching: 25.6Tbps". Sixty-four ports at 400 Gb/s is 25.6 Tb/s in one direction. Both figures are correct, they describe one switch, and the marketing page uses the larger one.

The same vendor then flips convention within one product family: Quantum-X800's 144 ports at 800 Gb/s are published as 115.2 Tb/s, which is the unidirectional arithmetic. And Dell sidesteps it entirely — Dell's Z-series overview publishes forwarding rate in packets per second and no capacity figure at all. When two spec sheets disagree by exactly 2×, nobody is lying to you; they are counting differently.

The Ethernet lineup: NVIDIA Spectrum and Dell PowerSwitch

Dell resells the NVIDIA Spectrum Ethernet line as PowerSwitch SN-series, and sells its own Z and S series alongside it. The models are not interchangeable, and the differences that matter to a fabric buyer are connector type, breakout ladder and power topology — not headline speed. Ports, power and capacity below are from NVIDIA's SN5000, SN4000 and SN2201 hardware manuals; the ASIC generations are from Lenovo Press's Spectrum switch product guide.

Model	ASIC generation	Ports	Connector	Capacity	Typical power	RU
SN2201	Spectrum	48 × 1GbE + 4 × 100GbE	RJ45 + QSFP28	448 Gb/s	98 W	1U
SN4700	Spectrum-3	32 × 400GbE	QSFP-DD	12.8 Tb/s	630 W	1U
SN5400	Spectrum-4	64 × 400GbE + 2 × 25GbE	QSFP-DD + SFP28	25.6 Tb/s	670 W	2U
SN5600D	Spectrum-4	64 × 800GbE + 1 × 25GbE	OSFP + SFP28	51.2 Tb/s	940 W	2U
SN5610	Spectrum-4	64 × 800GbE + 2 × 25GbE	OSFP + SFP28	51.2 Tb/s	900 W passive; 2.08 kW with modules	2U

Two details in that table are routinely got wrong. The SN4700 is Spectrum-3, not Spectrum-4, and the SN4700 and SN5400 are QSFP-DD, not OSFP — the 400GbE tier splits across connector families inside one product line. Breakout ladders differ per model too: the SN5400 reaches 256 ports of 10/25/50/100GbE, while the SN5610 stops at 50GbE on the bottom rung where the SN5600 goes to 10GbE.

Power topology separates the 800GbE models more than their specifications suggest. The SN5600D has no power supplies at all — NVIDIA states it is "powered by a DC busbar assembly", making it a rack-architecture decision rather than a switch decision. The SN5610 is the only member of the family with four power supplies and five fans against two and four on its siblings: the practical difference between 1+1 and 2+2 redundancy. NVIDIA positions the family as "smart spine/super-spine" switches "ideally suited for both top-of-rack leaf and fixed configuration spines."

On buffering, NVIDIA's own two document classes disagree about the SN5000 series, and a buyer comparing sources will meet both numbers. The Spectrum SN5000 series datasheet describes "the fully shared 160 megabyte (MB) packet buffers" and lists 160 MB in its specification table for all four models; the SN5000 hardware manual introduction states "on-chip fully-shared 128MB packet buffering". Both are NVIDIA's own documents — no third party is involved in either figure. The SN4000 series carries no such split: 64MB appears in both the SN4000 datasheet specification table and the SN4000 hardware manual's "on-chip fully-shared 64MB packet buffering". What all four documents agree on is the architecture, which is the part that decides behaviour under bursty collective traffic: one shared pool any congested port can draw on, rather than memory segmented per port group. If a comparison hands you a Spectrum-4 buffer figure, check which NVIDIA document it came from.

Spectrum-X vs InfiniBand: Spectrum-X is a platform, not a switch

You cannot buy Spectrum-X by buying a switch. NVIDIA describes Spectrum-X as coupling "the NVIDIA Spectrum-X Ethernet switch and the NVIDIA Spectrum-X Ethernet SuperNIC" — both halves, working together.

The mechanism explains why. Spectrum-X uses fine-grained adaptive routing that load-balances packet by packet across the network rather than per flow. That deliberately delivers packets out of order, so "the NVIDIA BlueField-3 SuperNIC makes sure to re-order the packets and place them in the host memory." Congestion control runs on high-frequency telemetry probes with flow metering. Buy the switch, keep third-party NICs, and the packet-spraying half has nothing at the endpoint to put the packets back in order — you have an ordinary Ethernet switch with a premium name. In an NVIDIA Spectrum-X vs InfiniBand comparison, that is the sentence that matters most: the comparison is only valid if both halves of the platform are in the build.

Dell-silicon leaf and spine: the Z and S series

Where the SN-series is NVIDIA silicon in a Dell badge, the Z and S series are Dell's own switches — and they are where an AI-adjacent build meets a conventional data-centre network. Dell describes the Z9864F-ON in its own words as "a two rack-unit (U), fixed switch with 64 800GbE ports based on the OSFP112 form factor, and 2 10GbE SFP+ ports", and publishes forwarding rates of up to 20.3 billion packets per second for it, 10.2 Bpps for the 400GbE Dell PowerSwitch Z9664F-ON and 5.2 Bpps for the 1RU Z9432F-ON.

The S5448F-ON is the practical entry point: 48 × 100GbE SFP56-DD plus 8 × 400GbE QSFP56-DD and 2 × 10GbE SFP+ in 1RU, 920 W maximum and 250 W typical, two power supplies in 1+1 with six hot-swappable fan trays, rated to 45 °C. It runs Enterprise SONiC Distribution by Dell Technologies or Dell SmartFabric OS10 over ONIE. It is also the one switch in this whole range that can be bought online rather than quoted.

Cages, ports and the arithmetic that catches people out

On an InfiniBand switch, one physical cage carries two ports. The QM9700 and QM9790 present 64 ports of 400 Gb/s across 32 OSFP cages. The Q3400-RA presents 144 ports across 72 cages. Size a cable order off the port count and you can be wrong by a factor of two in either direction — over-ordering if you use twin-port cables, or planning 64 faceplate runs where only 32 openings physically exist.

The second-order rule is the one people get backwards: breakout changes the port count, not the bandwidth. A QM9700 configured as 128 ports of 200 Gb/s is still 25.6 Tb/s of silicon. Splitting an 800G OSFP cage into 2 × 400G or 4 × 200G divides the same capacity into more, slower links — useful for connecting more endpoints, useless for adding throughput.

Sizing leaf and spine from the rail count

In a rail-optimised design, the number of leaf switches equals the number of GPUs per node. That single sentence does most of the sizing work, and NVIDIA's published counts let you check the arithmetic end to end.

NVIDIA's DGX SuperPOD reference architecture defines a scalable unit as 32 nodes with 8 leaf and 4 spine switches, rail-aligned so that traffic on a rail is "always one hop away from the other 31 nodes". Work it through:

1. Eight GPUs per node means eight rails, so eight leaf switches per scalable unit — one per rail.
2. Thirty-two nodes each contribute one port on that rail: 32 downlinks per leaf.
3. A 64-port leaf therefore has 32 uplinks left — a 1:1, non-blocking design.
4. Eight leaves × 32 uplinks = 256 uplinks per unit; four units = 1,024 uplinks; 1,024 ÷ 64 ports per spine = 16 spine switches and 1,024 spine-to-leaf cables.

NVIDIA's own table for a full four-unit deployment gives 32 leaf, 16 spine and 1,024 spine-to-leaf cables. The derivation matches exactly, which means you can apply the same arithmetic to a cluster that is not a SuperPOD:

Leaf switches = one per rail. Downlinks per leaf = nodes in the unit. Uplinks per leaf = leaf port count − downlinks. Oversubscription = downlinks : uplinks.

On a 64-port leaf, 32 nodes is the maximum at 1:1. Accept 2:1 and the same leaf serves 42 nodes; accept 3:1 and it serves 48. You are trading spine ports for node count, and on an all-reduce-heavy training job you pay for that trade in collective time. The topology this produces at two levels is a fat tree topology — the same structure used in HPC for decades, arrived at from the rail count rather than from theory.

One detail from that reference architecture is worth designing around: a full four-unit SuperPOD is 127 nodes, not 128, because "a DGX system must be removed to accommodate for UFM connectivity." You give up a node's worth of ports to the fabric manager. Plan for 127.

Cabling and optics: what will not seat

The most expensive mistake in an AI fabric order is not choosing the wrong switch. It is ordering optics that physically will not go into it. The 400G and 800G tiers split across four mechanically incompatible cage types inside a single vendor's range, and a spec table showing "400G" on both sides tells you nothing about whether the module fits.

Family	Model	Cage / connector	Accepts older QSFP optics?
Quantum-2 InfiniBand	QM9700 / QM9790	OSFP (twin-port, 2 × 400G per cage)	No
Quantum-X800 InfiniBand	Q3400-RA	OSFP (2 × 800G per cage)	No
Spectrum-3 Ethernet	SN4700	QSFP-DD	Yes — QSFP56, QSFP28, QSFP+
Spectrum-4 Ethernet	SN5400	QSFP-DD + SFP28	Yes
Spectrum-4 Ethernet	SN5600D / SN5610	OSFP + SFP28	No
Dell Z-series	Z9432F-ON / Z9664F-ON	QSFP56-DD + SFP+	Yes
Dell Z-series	Z9864F-ON	OSFP112 + SFP+	No
Dell S-series	S5448F-ON	SFP56-DD + QSFP56-DD + SFP+	Yes, on the QSFP side

OSFP vs QSFP-DD: the difference that decides your optic inventory

They are not interchangeable, and the asymmetry runs one way. Juniper — deliberately a non-NVIDIA source on a question NVIDIA has an interest in — states it without hedging: "QSFP-DD800 and OSFP800 are different form factors and aren't physically compatible." OSFP is the larger body, 60 connector pins, with "no backward compatibility with QDD optics". QSFP-DD is smaller, 38 pins, and is "compatible with some of the other QSFP transceiver module ports such as QSFP56, QSFP28, and QSFP+."

The buying consequence is concrete: a QSFP-DD switch can reuse an existing 100G or 200G optic inventory; an OSFP switch cannot reuse anything. If you hold a large installed base of QSFP optics, that belongs in the switch decision, not in the cabling stage after it.

The OSFP heat-sink trap on NDR cables

An NDR InfiniBand cable has two different OSFP form factors — one on each end — and they are not interchangeable. Quantum-2 switch cages take a finned-top module that carries its own heatsink; a ConnectX-7 host adapter takes a flat-top module, because the adapter's cage supplies a riding heatsink and the module has to be smooth to slide under it.

Vendor part listings say so in the product title itself — for example an 800G OSFP finned-top to 2 × 400G OSFP flat-top NDR breakout cable, described as being for Quantum-2 switches and ConnectX-7 adapters. Juniper documents the two heat-sink classes independently: OSFP-IHS at "13 mm height for OSFP within the cage" against OSFP-RHS at "9.5 mm height without heat sink". Order the same type on both ends and the cable will not seat. It is invisible in a specification table, it is a returns event, and it only shows up when somebody is physically holding both ends.

Reach, and why the cable budget has a power line

The jump from 400G to 800G roughly halves how far passive copper carries a signal. Typical reach for passive DAC runs around three metres at 400G and around two at 800G; active copper extends that to roughly three to five metres, active electrical cables with retimers to five to nine, and active optical cable to about a hundred. Those are industry-typical figures rather than a manufacturer specification and should be confirmed against the specific part.

The consequence people miss is electrical rather than mechanical. A rack layout that worked on passive DAC at 400G may need active cable at 800G, and every active cable draws power at both ends — Juniper puts an 800G gray optic at "16 W to 18 W approximately" and ZR or ZR+ optics up to 30 W. On a switch already drawing 900 W typical, a fully populated faceplate of active optics is a second power budget on top of the first. Budget the cable power, not just the switch power.

It is also why our configurator offers each switch only the cable and transceiver types its own cages accept — the QM9700 gets NDR InfiniBand cabling and nothing else, the 1GbE SN2201 gets neither — with the cable groups presented as checkbox lists and each cable type you tick taking its own quantity, so a 32-cage order can be a real mixed bill of materials rather than one cable choice per switch. If you are sourcing separately, our optical transceivers and DAC cables category carries the wider range.

Common mistakes when specifying an AI fabric

Most fabric problems are specification errors, not hardware failures. These are the ones that recur, each with the symptom it produces:

1. Sizing the cable order from the port count on an InfiniBand switch. Count cages: 32 on a QM9700, 72 on a Q3400-RA. Symptom: a two-fold over-order, or a rack plan with more cable runs than faceplate openings.
2. Buying an externally managed switch as the only switch in the fabric. Symptom: the fabric never comes up, and there is no console port to diagnose it from.
3. Ordering the same OSFP heat-sink type on both ends of an NDR cable. Symptom: the cable will not seat at one end.
4. Mixing OSFP and QSFP-DD. Symptom: the module physically does not enter the cage. Match the connector, not the speed.
5. Comparing two switching-capacity figures without checking the duplex convention. Symptom: a switch appears twice as capable as an identical one. Divide by ports × port speed first.
6. Ordering the wrong airflow variant. Symptom: the switch draws hot-aisle air — and the two variants are not even rated for the same room temperature.
7. Designing a four-unit pod for 128 nodes. Symptom: you get 127. One system's ports go to fabric management.
8. Budgeting one network. Symptom: compute fabric purchased, storage and management networks unfunded halfway through deployment.
9. Assuming 400G copper reach carries to 800G. Symptom: passive runs that worked in the old rack layout will not link in the new one.
10. Expecting a 1GbE management switch to carry AI traffic. Symptom: a switch bought for the compute fabric that is architecturally an access switch.
11. Planning to reuse existing 100G or 200G optics on OSFP ports. Symptom: an entire optic inventory stranded by a connector decision made elsewhere.
12. Buying Ethernet switches and expecting InfiniBand behaviour out of the box. Symptom: RDMA works in a lab test and collapses under congestion, because PFC, ECN and traffic-class mapping were never aligned across switch, NIC, driver and host.

How to test a fabric before you commit to it

Test the fabric under the workload it will actually run, under congestion, with something deliberately broken. A fabric that passes a quiet point-to-point bandwidth test tells you almost nothing about how a training job will behave on it. Ten measurements are worth taking, in this order:

1. GPU utilisation during a real collective operation, not at idle and not on a synthetic stream. If GPUs are waiting on the network, this is where it shows.
2. Node-to-node RDMA latency at small message sizes — the regime collectives actually live in, and the one where fabric differences are largest.
3. Achieved bandwidth per rail against line rate, per rail rather than aggregate, so one badly-behaved path cannot hide in an average.
4. Scaling efficiency from one node to two to four to a full scalable unit. The shape of that curve is the fabric's real report card.
5. CPU utilisation attributable to network work. If RDMA is working, this stays low; if it silently fell back, it will not.
6. Interface error counters — and on Ethernet, PFC pause counters and ECN marks specifically. Pause storms are the classic lossless-Ethernet failure and they are invisible unless you look.
7. Packet loss under deliberate congestion. Load the fabric past comfort on purpose. The best case tells you nothing you need to know.
8. Wall-clock time for one real training step, compared across configurations. This is the number the business is actually buying.
9. Stability across a multi-hour run. Thermal and buffer problems are time-dependent and will not appear in a five-minute test.
10. Behaviour when you fail something on purpose — pull a port, then a whole switch. On InfiniBand, also fail the subnet manager and confirm the fabric survives it.

Two more that are specific to buying rather than operating: seat one of every optic type you have ordered before racking the whole delivery — a 32-cage switch is 32 opportunities to discover a form-factor mismatch late — and confirm the link trains at the rated speed rather than negotiating down, which is the usual first sign of a marginal cable.

Buying AI fabric switches on the secondary market

AI fabric hardware reaches the secondary market the same way everything else does: generation succession, cluster decommissions and refresh cycles. When a newer switch generation ships, the previous one starts coming out of clusters built when it was current — and at this tier those units were specified to run continuously in a controlled environment, which is materially better provenance than most used hardware carries.

The market is thinner at the top than at the bottom, and it is worth being straight about where the line falls. Hundred-gigabit hardware circulates freely; the 400G and 800G tiers circulate far less, which is why every switch in this range is quoted rather than priced, and why the cables and transceivers for them are quoted too. What we carry, we carry as configurable built-to-order units on a five-to-seven business day lead time: the internally managed Quantum-2 QM9700, the externally managed QM9790, the Quantum-X800 Q3400-RA, and the Spectrum-based PowerSwitch SN2201, SN4700, SN5400, SN5600D and SN5610, with the Dell-silicon Z9432F-ON, Z9664F-ON, Z9864F-ON and S5448F-ON alongside them in the same refurbished Dell network switches and AI fabric range. Those switches sit together, with a side-by-side spec comparison, on our AI networking and high-speed switch lineup.

What transfers with the box, and what does not

On open-networking hardware, the switch and the software it runs are separate purchases, and the image installed when it arrives tells you nothing about what you are licensed to run. Every Dell Z- and S-series switch boots ONIE and can run Enterprise SONiC Distribution by Dell Technologies or Dell SmartFabric OS10; the SN-series runs Cumulus Linux or Enterprise SONiC. A used unit may arrive with any of them installed, in any state.

Four questions settle it before money moves:

1. What network operating system is licensed — not installed — and is that licence transferable to you? This is the single largest hidden cost in a used open-networking purchase.
2. On InfiniBand, where does the subnet manager come from in your design? If the answer is "the switch", you need the internally managed model. If it is UFM or a host, ask what fabric-management licensing is included and whether it moves with the hardware.
3. What support entitlement, if any, is attached? Manufacturer support on used hardware is a separate conversation from the sale, and it is better had before it.
4. What warranty comes with the unit itself? Every AI fabric switch we list carries a warranty selection made in the configurator — two, three or five years — because a switch with no coverage is a different product from the same switch with five years on it.

What to check on arrival

1. Airflow direction against your aisle. Vendor part-number listings consistently map the -NS2F suffix to power-to-connector (forward) airflow and -NS2R to connector-to-power (reverse). There is a second consequence beyond orientation: NVIDIA's specification table rates the two differently, at 0–35 °C for forward airflow and 0–40 °C for reverse. If the room runs warm, the airflow variant is a thermal decision, not just a direction.
2. Cage count and cage type against your optic order. Confirm 32 cages on a Quantum-2, 72 on a Q3400-RA, and that the modules you hold match OSFP, QSFP-DD, QSFP56-DD or OSFP112 as appropriate.
3. Management interfaces physically present. On a Quantum-2, a management port, console port and front USB port mean you have the internally managed model. Their absence means you have the externally managed one — whatever the listing said.
4. Power supply count and type. Two PSUs and seven fans on a Quantum-2; four PSUs and five fans on an SN5610; no PSUs at all on an SN5600D, which is DC busbar only and cannot be racked in an AC-only cabinet.
5. Input voltage. The Quantum-2 switches are 200–240 Vac only. That is a real constraint in facilities wired for lower-line single-phase power in North America, and it is better discovered before delivery.
6. Firmware and configuration state. Clear the prior owner's configuration, confirm the firmware level, and — on InfiniBand — confirm which subnet-manager role the unit was last playing.
7. Rails, cables and accessories as separate line items. Rail kits, power cords and console cables are ordered, not assumed.

Every switch we list goes through receiving, testing and configuration before it is sold, and the photography on our product pages is of hardware that came through that process rather than a manufacturer render. If you are buying elsewhere, the list above is still the list.

A pre-purchase checklist

Answer these before you place the order. Any one of them that cannot be answered is a design decision that has not been made yet.

1. How many GPU nodes, and how many GPUs per node? The second number sets the leaf count.
2. Training, inference, or both? Collectives decide whether in-network compute is worth paying for.
3. What oversubscription ratio is acceptable on the compute fabric, and who signed off on it?
4. Where does the subnet manager live — on a switch, in UFM, or on a host?
5. Which connector family are you standardising on, and does it strand any optics you already own?
6. What are the longest cable runs in the rack plan, and does passive copper reach at your port speed?
7. Have you budgeted the storage network, the in-band management network and the out-of-band management network as separate purchases?
8. Which airflow variant matches the aisle, and what is the room's actual ambient temperature at the top of the rack?
9. Does the facility supply the input voltage the switch requires, at the circuit capacity a fully populated faceplate of active optics will draw?
10. Who operates the fabric on day two — and if it is Ethernet, who has run PFC and ECN in production before?
11. What network operating system will run, and is the licence for it included and transferable?
12. What is the warranty position on every item in the order, including cables and optics?

Frequently Asked Questions

What is the difference between the NVIDIA QM9700 and QM9790?

They are the same 1U switch — 64 ports of 400 Gb/s NDR InfiniBand across 32 OSFP cages, two power supplies, seven fans, identical chassis. The difference is management. The QM9700 is internally managed: it carries an on-board subnet manager that NVIDIA rates for bringing up "up to 2,000 nodes", running MLNX-OS with CLI, WebUI, SNMP and JSON interfaces. The QM9790 is externally managed and has no console port, no management port and no USB port — it expects NVIDIA Unified Fabric Manager, or a host-based subnet manager, elsewhere in the fabric. The management hardware also shows up in the power draw: 747 W typical for the QM9700 against 640 W for the QM9790.

Does an InfiniBand fabric need a subnet manager?

Yes. An InfiniBand fabric is not self-forming — a subnet manager discovers the topology, assigns local identifiers and programs the forwarding tables before any traffic flows. You can get one in three ways: on board an internally managed switch such as the QM9700, from NVIDIA UFM running centrally, or from a host running a software subnet manager. If every switch you buy is externally managed and nothing else in the design runs one, the fabric will not come up — and on a QM9790 there is no console port to fix it from locally.

How many ports does a QM9700 actually have — 32 or 64?

64 ports across 32 physical cages. Each OSFP cage on a Quantum-2 switch is a twin-port carrying two independent 400 Gb/s NDR links, so NVIDIA counts 64 ports while the faceplate shows 32 openings. The same rule scales: the Quantum-X800 Q3400-RA is 144 ports over 72 OSFP cages, and the Q3200-RA is 72 ports over 36. Size a cable order off the port count and you can be wrong by half — count cages, then decide how each one is being used.

Is InfiniBand proprietary, and who owns it?

The specification is not proprietary. InfiniBand is maintained by the InfiniBand Trade Association, whose steering committee includes HPE, IBM, Intel and NVIDIA, and it was the IBTA — not NVIDIA — that published the XDR 800 Gb/s specification. What is concentrated is supply: NVIDIA acquired Mellanox, the last independent InfiniBand supplier, and in practice NVIDIA is where InfiniBand switch silicon comes from. Open standard, concentrated supply — those are different risks, and they are worth separating when you assess lock-in.

What is the difference between OSFP and QSFP-DD, and can I use one in the other?

No. Juniper states plainly that "QSFP-DD800 and OSFP800 are different form factors and aren't physically compatible." OSFP uses a 60-pin connector in a larger body; QSFP-DD uses 38 pins in a smaller one. The practical difference for a buyer is backward compatibility: QSFP-DD ports accept older QSFP56, QSFP28 and QSFP+ optics, while OSFP accepts none. The split runs straight through the 400G tier — the SN4700 and SN5400 are QSFP-DD, the Quantum switches and the SN5600 family are OSFP, Dell's Z9432F-ON and Z9664F-ON are QSFP56-DD, and the Z9864F-ON is OSFP112.

How much lower is InfiniBand latency than Ethernet, really?

Less than most comparisons imply, because published figures are measured at three different layers. Switch port-to-port latency is in the hundreds of nanoseconds on both fabrics — Dell publishes it for its Ethernet switches, and NVIDIA does not publish a port-to-port figure for Quantum-2 at all, which is exactly why third-party pages quote it four different ways without agreeing. End-to-end RDMA latency, which is what an application actually experiences, is low single-digit microseconds on both, with InfiniBand ahead. Figures in the tens of microseconds quoted as "Ethernet" are TCP/IP without RDMA, which is neither fabric's RDMA path. Compare like with like or the numbers are meaningless.

What do the NS2F and NS2R suffixes mean on an NVIDIA switch part number?

They denote airflow direction. Vendor part-number listings consistently map -NS2F to power-to-connector airflow, described as forward, and -NS2R to connector-to-power, described as reverse. Get it wrong and the switch pulls hot-aisle air. There is a second consequence worth knowing: NVIDIA's own specification table rates the two directions differently — forward airflow is rated 0–35 °C ambient and reverse airflow 0–40 °C. If your room runs warm, the airflow variant is not just an orientation decision.

Can you train large models on RoCE Ethernet instead of InfiniBand?

Yes, and the largest publicly documented example does. Meta has stated that it uses "RoCEv2 as the inter-node communication transport for the majority of our AI capacity" and has trained Llama 3 on a RoCE cluster built from standard Ethernet silicon. What that account also shows is where the work is: Meta ran "without DCQCN for our 400G deployments", relying on priority flow control alone, and found that routing decisions rather than hardware cost the performance — path-pinning degraded training "up to more than 30%" and better load balancing recovered "up to 40% for the AllReduce collective." RoCE is viable at scale. It is a tuning project, not a purchase.

How many leaf and spine switches does a 32-node GPU pod need?

In NVIDIA's DGX SuperPOD reference architecture, a 32-node scalable unit uses eight leaf switches and four spine switches on the compute fabric. The leaf count comes from the rail count: each node has eight GPUs and therefore eight compute links, and each rail gets its own leaf, so every node is one hop from the other 31 on that rail. Each 64-port leaf then carries 32 downlinks and 32 uplinks — a 1:1, non-blocking design. Scale it and the arithmetic holds: four scalable units need 32 leaf and 16 spine switches and 1,024 spine-to-leaf cables. One detail worth planning around — a full four-unit deployment is 127 nodes, not 128, because one system's ports are given up to fabric management.

What does the "-RA" in Q3400-RA mean?

Air-cooled. NVIDIA describes the RA models as an "air-cooled system designed for standard 19-inch rack cabinets." It is not an airflow-direction token — that is the separate -NS2F and -NS2R suffix on the Quantum-2 line. The counterpart to RA is the liquid-cooled LD designation used on the Quantum-X800 switches built for direct liquid cooling.

If you know which fabric you are building, the next decision is the switch and where its subnet manager lives: start with the internally managed NVIDIA Quantum-2 QM9700 or the externally managed QM9790 on the InfiniBand side, or the SN5610 on the Ethernet side. If you are still weighing the options, send us the node count, the GPUs per node and the longest cable run in the rack plan, and we will quote the fabric — switches, optics and cables — against the arithmetic in this guide.