How do you tell if a GPU cluster actually works?

The racks are in. The cables are run. The vendor sends photos of blinking lights and asks for the rest of the invoice. None of that tells you whether the cluster works.

I’ve built infrastructure and consulted on GPU cluster deals, and “works” is never a yes or no. It’s a set of measured numbers, each with a public formula behind it. Any of them can come in wrong while every rack powers on and every GPU shows up in nvidia-smi. A fabric wired at 3:1 oversubscription instead of 1:1 looks the same from the front of the rack. A GPU that throttles under sustained load looks fine on a five-minute smoke test. Storage with lots of capacity but not enough throughput looks fine too, until a job is running.

That gap between “installed” and “performs” is where non-technical buyers get taken: investors, lawyers, family offices, first-time operators. You don’t need to become an infrastructure engineer to close it. You need to know which six numbers to demand, how each is computed from public specs so nobody can hand-wave it, and what to write into the contract so the last payment depends on the measured number, not the install date.

1. Bisection bandwidth: what the fabric is supposed to hit

Cut the cluster in half. Have every GPU in one half talk to a GPU in the other half at the same time. The total data rate across that cut is the bisection bandwidth. It’s the standard stress case for a training fabric, because collective operations like all-reduce move data across the whole cluster at once, not node to node.

The theoretical figure is arithmetic, not a vendor claim. Take a reference cluster of 1,152 GPUs, each with an 800 Gb/s scale-out network port. That’s the class of port NVIDIA’s ConnectX-8 SuperNIC and Quantum-X800 InfiniBand platform were built around when NVIDIA announced them on March 18, 2024. In a fully non-blocking fabric, the bandwidth across the cut equals the total port bandwidth of the smaller half:

• Total endpoint bandwidth: 1,152 x 800 Gb/s = 921,600 Gb/s (921.6 Tb/s)
• Split into two halves of 576 GPUs each: 576 x 800 Gb/s = 460,800 Gb/s (460.8 Tb/s)
• Converted from bits to bytes: 460,800 Gb/s / 8 = 57,600 GB/s, or 57.6 TB/s

So 57.6 TB/s is the ceiling for this port count and speed. Rerun the same division with whatever GPU count and NIC speed your vendor quotes.

No real fabric hits 100 percent of that, and you shouldn’t demand it. Encoding overhead, switch latency and uneven load balancing across rails all take a bite. Together AI published a practitioner’s guide to running large GPU clusters on August 13, 2024 with a concrete example: a healthy fabric showing all-reduce bandwidth around 92 percent of the theoretical maximum, about 370 GB/s against a 400 GB/s fabric. Use something in that neighborhood, roughly 90 percent of your own computed ceiling, as the acceptance threshold. Below that, something in the wiring, firmware or switch config is costing you real throughput, permanently.

Theoretical ceiling57.6

~90% acceptance floor51.84

90 percent of 57.6 TB/s is 51.84 TB/s. Put that number in the contract. Not “the fabric should perform well.”

2. NCCL busbw vs. algbw: the spec sheet speed is not what you measure

Here’s the trap. The 800 Gb/s on the spec sheet is a link-layer number. It’s not what a training job sees. A vendor quoting it as “the fabric speed” isn’t lying, they’re quoting the wrong layer.

The standard way to measure what a fabric delivers is NCCL’s own test suite, nccl-tests. It reports two numbers for the same run:

• algbw (algorithm bandwidth): data size divided by time, S / t.
• busbw (bus bandwidth): algbw corrected for how much data actually crosses the wire for that collective.

NCCL’s performance doc (nccl-tests, doc/PERFORMANCE.md, NVIDIA’s GitHub repository) gives the correction factors. For all-reduce, which syncs gradients across every GPU in a training job:

busbw = algbw x 2(n-1)/n

For all-gather, which reassembles sharded data:

busbw = algbw x (n-1)/n

At the reference cluster’s scale, n = 1,152 GPUs:

• All-reduce factor: 2 x (1,152 - 1) / 1,152 = 2,302 / 1,152 ≈ 1.998, effectively 2x.
• All-gather factor: (1,152 - 1) / 1,152 = 1,151 / 1,152 ≈ 0.999, effectively 1x.

Put plainly: at this scale, syncing one byte of gradient costs close to two bytes of wire traffic for all-reduce, and close to one byte for all-gather. That factor is already inside busbw. So ask for the measured busbw from an all_reduce_perf run across the full node count, not the NIC’s rated speed. It should land within roughly 90 percent of the bisection bandwidth you computed in section 1, the same 370-against-400 GB/s kind of result Together AI published.

3. Where oversubscription hides: fat tree wiring and rail alignment

InfiniBand HPC fabrics are almost always a fat tree. Leaf switches connect to the GPUs and NICs, spine switches connect the leaves, and the biggest builds add a core tier above the spines. Introl’s network topology write-up from March 16, 2026 describes the non-blocking version plainly: “aggregate bandwidth between any two halves of the cluster equals the total bandwidth into either half,” which is the same bisection arithmetic as section 1.

The oversubscription ratio is what hides. A leaf switch with, say, 64 ports can be wired two very different ways:

• Non-blocking (1:1): 32 ports down to compute, 32 ports up to the spine. Every unit of downlink has a matching unit of uplink.
• Oversubscribed (3:1): 48 ports down to compute, only 16 up to the spine. 48 units of downlink traffic fight over 16 units of uplink, so at most a third of it can cross to another leaf at once.

Same switch. Same port count on the spec sheet. Same cable count in the rack photo. The only difference is which ports do what. The Introl piece notes that plenty of general-purpose deployments accept 2:1 as a cost trade-off. Training clusters usually can’t. Collectives are all-to-all by design, every GPU has to reach every other GPU, so one bottleneck anywhere in the tree throttles the whole job.

Rail-optimized wiring adds a second way to fail. Multi-GPU nodes usually give each GPU its own NIC, and the network is cabled so that, for example, “NIC 3 on every node” forms its own plane, or rail, through its own leaf switches. That shortens the path between a GPU and its counterpart on another node. It only works if the cabling actually keeps that alignment. One cable patched into the wrong rail during install, and you have a fabric that passes a basic connectivity check while quietly adding hops to every collective that crosses rails.

Ask for the leaf-to-spine port ratio in writing, and ask for the rail map. Neither shows up in a rack photo. Both can be checked from the switch config in minutes by anyone who knows to ask.

4. A cluster is only as fast as its worst GPU

A synchronous training job runs at the speed of its slowest GPU. Every GPU in a collective waits for every other one before the next step. So one throttled or flaky GPU out of a thousand doesn’t crash anything. It just caps every job scheduled onto it, forever, and never shows up as a failure on a dashboard that only checks whether nodes are reachable.

Two tools catch most of this before acceptance.

DCGM diagnostics. NVIDIA’s Data Center GPU Manager runs tiered tests, from a quick deployment check (run level 1, seconds) up to extended stress and memory-pattern testing (run level 3 to 4, minutes to tens of minutes per node, per NVIDIA’s DCGM user guide). A vendor who can only show a run-level-1 pass has checked that the lights turn on. Ask for run level 3 or higher, on every node, not a sample.

Xid error logs. Xid messages are the NVIDIA driver’s own error reports. A public field guide to them (updated March 13, 2026, published April 21, 2026) lists the codes that matter most at cluster scale. Xid 79, “GPU has fallen off the bus,” is fatal: the GPU drops off the PCIe device list and needs someone to physically fix it. Xid 63 is ECC page retirement. One is normal, but the guide says a GPU getting close to half of its 64-page retirement limit is trending toward failure, and you want to know before it dies mid-job. Xid 48 is a double-bit ECC error, live data corruption. That should stop a burn-in cold.

Together AI’s guide (August 13, 2024) also covers catching thermal throttling: run a sustained load generator like gpu-burn for a long stretch, not a few minutes, and log temperature and power draw. A GPU that’s fine on a short test can still overheat and throttle once it’s carrying an hours-long job. Ask for the burn-in duration in hours, not just confirmation that a burn-in happened.

5. Storage that can feed the GPUs, not just hold the data

Usable capacity in petabytes tells you how much data you can keep. It says nothing about whether storage can hand that data to the GPUs fast enough to keep them busy. That’s throughput, in gigabytes per second, and people quote the two as if they were the same thing. They’re not.

Hammerspace’s storage architecture write-up from June 25, 2026 gives a public per-GPU guideline: “modern GPUs like the NVIDIA H100 can consume data at 3 to 4 GB/s per GPU.” Apply that to a modest 64-GPU training pod, a common single-tenant unit inside a bigger cluster:

• Low end: 64 x 3 GB/s = 192 GB/s aggregate sustained throughput
• High end: 64 x 4 GB/s = 256 GB/s aggregate sustained throughput

That’s what storage has to sustain, continuously, under real data loading and checkpointing. Not the peak from the marketing sheet, and not the capacity on the purchase order. Lots of capacity with too little throughput starves the GPUs the same way an undersized fabric does. The compute you paid for sits idle waiting on data, and none of that shows up when someone checks that the drives are mounted.

6. What to put in the contract

Every number above can be checked before the final invoice is paid. So write them into the contract as conditions of payment, not hopes. An acceptance clause worth signing should spell out, at minimum:

1. The exact tests and tools, named, not described: nccl-tests all-reduce and all-gather runs at full node count, DCGM diagnostics at a stated run level (3 or higher) on every node, a stress test such as gpu-burn for a stated number of hours, and a storage throughput benchmark at a stated block size and concurrency.
2. The pass thresholds, computed the way sections 1 through 5 compute them, not asserted: theoretical bisection bandwidth for the port count and speed actually delivered, a busbw target near 90 percent of that ceiling, zero fatal Xid events over the burn-in window, and a sustained storage throughput figure in GB/s per GPU.
3. The duration, in hours. A fabric or a GPU can pass a five-minute test and fail an eight-hour one. Hours, not minutes, is what catches thermal throttling and intermittent hardware faults.
4. Who runs it and who watches, by name. Raw logs reviewed by the buyer’s own technical person, or the test run in front of them. Not a summary PDF the vendor made on its own.
5. A holdback tied to the result, not the install date. A common structure holds back a defined minority of the contract value until acceptance passes, with a stated cure period and re-test procedure if it doesn’t. The exact percentage and cure period are a negotiation, and I can’t hand you those. The principle doesn’t move: racks arriving in the room is a delivery milestone, not a payment milestone.

The order to check things in

1. Compute the theoretical bisection bandwidth yourself from the delivered port count and port speed, and demand a measured busbw around 90 percent of it.
2. Ask for the measured busbw from an all_reduce_perf run, not the NIC’s rated link speed. They’re different numbers on purpose.
3. Ask for the leaf-to-spine port ratio in writing. A switch with the right port count can still be cabled at 2:1 or 3:1.
4. Ask for a DCGM run-level 3+ diagnostic and an hours-long burn-in log across every GPU, not a sample, and check it for Xid 79, 63, and 48 events.
5. Ask for measured, sustained storage throughput in GB/s per GPU, not usable capacity in petabytes.
6. Put all five of the above into the contract as the acceptance gate, with a defined holdback and a re-test path, so the final payment depends on the cluster that was measured, not the one that was promised.

You don’t have to be an infrastructure engineer for any of this. You have to know the six numbers exist, know how each one is computed from information the vendor already has, and refuse to release the balance until someone hands you the measured version of all six. For what that same cluster costs to run once it passes, see what it actually costs to run a GPU cluster.