How to tell if a GPU cluster actually works

The racks are installed. The cables are run. The vendor sends photos of blinking lights and asks for the balance of the invoice. None of that tells you whether the cluster works.

“Works” for a GPU cluster is not a binary. It is a set of measured numbers, each with a public formula behind it, and each of them can come in quietly wrong while every rack still powers on and every GPU still shows up in nvidia-smi. A fabric that is wired at 3:1 oversubscription instead of 1:1 looks identical from the front of the rack. A GPU that throttles under sustained load looks identical to a healthy one on a five-minute smoke test. Storage that has plenty of capacity but not enough throughput looks identical to storage that can actually feed the job, until the job is running.

This is the gap between “the racks are installed” and “the cluster performs,” and it is exactly where a non-technical buyer, an investor, a lawyer, a family office, a first-time operator, gets taken. You do not need to become an infrastructure engineer to close that gap. You need to know which six numbers to demand, how each one is computed from public specs so nobody can wave a hand at it, and what to write into the contract so the final payment depends on the measured number rather than the installation date.

1. Bisection bandwidth: the number the fabric is supposed to hit

Bisection bandwidth is the total data rate available if you cut the cluster in half and every GPU in one half tries to talk to a GPU in the other half at the same time. It is the standard stress case for a training fabric because collective operations like all-reduce move data across the whole cluster at once, not node to node.

The theoretical figure is arithmetic, not a vendor claim. Take a reference cluster of 1,152 GPUs, each with an 800 Gb/s scale-out network port, the class of port NVIDIA’s ConnectX-8 SuperNIC and Quantum-X800 InfiniBand platform were built around when NVIDIA announced them on March 18, 2024. In a fully non-blocking fabric, the bandwidth available across the cut equals the total port bandwidth of the smaller half:

• Total endpoint bandwidth: 1,152 x 800 Gb/s = 921,600 Gb/s (921.6 Tb/s)
• Split into two halves of 576 GPUs each: 576 x 800 Gb/s = 460,800 Gb/s (460.8 Tb/s)
• Converted from bits to bytes: 460,800 Gb/s / 8 = 57,600 GB/s, or 57.6 TB/s

That 57.6 TB/s is the theoretical ceiling for this specific port count and speed, and you can rerun this exact division with whatever GPU count and NIC speed a vendor actually quotes you.

No real fabric hits 100 percent of that ceiling, and you should not demand that it does. Encoding overhead, switch latency, and imperfect load balancing across parallel network rails all take a bite. A practitioner’s guide to running large GPU clusters published by Together AI on August 13, 2024 gives a concrete worked example: a healthy fabric showing all-reduce bandwidth around 92 percent of the theoretical maximum, about 370 GB/s delivered against a 400 GB/s fabric. Treat a threshold in that neighborhood, call it roughly 90 percent of your own computed ceiling, as the number to write into the acceptance test. Below that, something in the fabric, wiring, firmware, or switch configuration, is costing you real, permanent throughput.

90 percent of 57.6 TB/s is 51.84 TB/s. That is the number to put in the contract, not “the fabric should perform well.”

2. NCCL busbw vs. algbw: why the port speed on the spec sheet is not the number you measure

Here is the trap: the 800 Gb/s figure on the previous section’s spec sheet is a link-layer number. It is not what a training job actually experiences, and a vendor quoting it as “the fabric speed” is not lying, they are quoting the wrong layer.

The standard benchmark for measuring what a fabric actually delivers is NCCL’s own test suite, nccl-tests, and it reports two different numbers for the same run:

• algbw (algorithm bandwidth): simply data size divided by time, S / t.
• busbw (bus bandwidth): algbw corrected for how much data actually has to cross the wire for that specific collective operation.

NCCL’s own performance documentation (nccl-tests, doc/PERFORMANCE.md, NVIDIA’s GitHub repository) gives the correction factors directly. For all-reduce, the operation used to synchronize gradients across every GPU in a training job:

busbw = algbw x 2(n-1)/n

For all-gather, used to reassemble sharded data:

busbw = algbw x (n-1)/n

Run the arithmetic at the reference cluster’s scale, n = 1,152 GPUs:

• All-reduce factor: 2 x (1,152 - 1) / 1,152 = 2,302 / 1,152 ≈ 1.998, effectively 2x.
• All-gather factor: (1,152 - 1) / 1,152 = 1,151 / 1,152 ≈ 0.999, effectively 1x.

In plain terms: at this scale, synchronizing one byte of gradient data across the whole cluster costs close to two bytes of actual wire traffic for all-reduce, and close to one byte for all-gather. That factor is baked into busbw before you ever compare it to a link-speed number. The number a buyer should ask for is the measured busbw from an all_reduce_perf run across the full node count, not the NIC’s rated speed, and it should be within roughly 90 percent of the theoretical bisection bandwidth computed in section 1, using the same 370-against-400 GB/s style of worked example Together AI published.

3. Where oversubscription hides: fat tree wiring and rail alignment

InfiniBand HPC fabrics are almost always built as a fat tree: leaf switches connect to the GPUs and NICs, spine switches connect the leaves, and in the largest builds a core tier connects the spines. A March 16, 2026 network topology write-up from Introl describes the non-blocking version of this design plainly: “aggregate bandwidth between any two halves of the cluster equals the total bandwidth into either half,” which is exactly the bisection bandwidth arithmetic from section 1.

The oversubscription ratio is the part that hides. A leaf switch with, say, 64 total ports can be wired two very different ways:

• Non-blocking (1:1): 32 ports down to compute, 32 ports up to the spine. Every unit of downlink capacity has a matching unit of uplink capacity.
• Oversubscribed (3:1): 48 ports down to compute, only 16 up to the spine. 48 units of downlink traffic now compete for 16 units of uplink capacity, so at most a third of that traffic can cross to another leaf at the same instant.

Both configurations use the same switch, the same port count on a spec sheet, and the same cable count in a rack photo. The difference is entirely in which ports get used for what, and the Introl piece notes that many general-purpose deployments accept 2:1 oversubscription as an acceptable cost trade-off, while training clusters typically cannot, because collective operations are all-to-all by design: every GPU needs to reach every other GPU, so a bottleneck anywhere in the tree throttles the whole job, not just the traffic crossing that one switch.

Rail-optimized wiring adds a second failure mode. Multi-GPU nodes typically dedicate one NIC per GPU, and the network is cabled so that, for example, “NIC 3 on every node” forms its own dedicated plane, or rail, through its own leaf switches. This shortens the path for GPUs that are supposed to talk to their counterpart rail on another node. It only works if the cabling actually preserves that alignment; a technician who patches a cable into the wrong rail during installation leaves you with a fabric that passes a basic connectivity check while quietly adding hops to every collective operation that crosses rails.

Ask for the leaf-to-spine port ratio explicitly, in writing, and ask for the rail map. Neither shows up in a rack photo, and both are checkable from the switch configuration in minutes by anyone who knows to ask.

4. A cluster is only as fast as its worst GPU

A synchronous training job runs at the speed of its slowest participant, because every GPU in a collective operation waits for every other one to finish before the next step starts. One throttled, degraded, or intermittently failing GPU out of a thousand does not crash the job. It just quietly caps every job scheduled onto it, indefinitely, without ever showing up as a hardware failure in a status dashboard that only tracks whether nodes are reachable.

Two tools do most of the work of catching this before acceptance:

DCGM diagnostics. NVIDIA’s Data Center GPU Manager runs a tiered set of tests, from a quick deployment check (run level 1, seconds) up through extended stress and memory-pattern testing (run level 3 to 4, minutes to tens of minutes per node, per NVIDIA’s DCGM user guide). A vendor who can only produce a run-level-1 pass across the fleet has run the equivalent of checking that the lights turn on. Ask for run-level 3 or higher, on every node, not a sample.

Xid error logs. Xid messages are the NVIDIA driver’s own error reports, and a public field guide to them (updated March 13, 2026, published April 21, 2026) catalogs the codes that matter most at cluster scale. Xid 79, “GPU has fallen off the bus,” is fatal: the GPU disappears from the PCIe device list entirely and needs hardware intervention. Xid 63 is ECC page retirement, individually normal, but the guide flags that approaching half of the 64-page retirement limit on a single GPU signals a card trending toward failure, worth flagging before it crosses the line mid-job. Xid 48 is a double-bit ECC error, a live data-corruption signal that should stop a burn-in test cold.

Together AI’s practitioner’s guide (August 13, 2024) also describes the practical version of catching thermal throttling before acceptance: running a sustained load generator such as gpu-burn for an extended period, not a few minutes, while logging temperature and power draw, because a GPU that looks fine on a short test can still overheat and throttle once it is carrying a real, hours-long job. Ask for the burn-in duration in hours, not the fact that a burn-in happened.

5. Storage that can feed the GPUs, not just hold the data

Usable capacity, measured in petabytes, answers a durability and retention question. It says nothing about whether the storage system can hand data to the GPUs fast enough to keep them busy, which is a throughput question, measured in gigabytes per second, and the two numbers are frequently quoted as if they were interchangeable.

A June 25, 2026 storage architecture write-up from Hammerspace gives a public per-GPU guideline: “modern GPUs like the NVIDIA H100 can consume data at 3 to 4 GB/s per GPU.” Apply that to a modest 64-GPU training pod, a common single-tenant unit within a larger cluster:

• Low end: 64 x 3 GB/s = 192 GB/s aggregate sustained throughput
• High end: 64 x 4 GB/s = 256 GB/s aggregate sustained throughput

That is the number the storage system needs to sustain, continuously, under real data loading and checkpointing, not the peak number from a vendor’s marketing sheet and not the raw capacity figure on the purchase order. A system with abundant capacity and insufficient throughput starves the GPUs exactly the way an undersized fabric does: the compute you paid for sits idle waiting on data, and nothing about that shows up when someone checks that the drives are present and mounted.

6. What to put in the contract

Every number above is checkable before the final invoice is paid, which means every one of them can be written into the contract as a condition of payment rather than a hope. An acceptance-test clause worth signing should specify, at minimum:

1. The exact tests and tools, named, not described: nccl-tests all-reduce and all-gather runs at full node count, DCGM diagnostics at a stated run level (3 or higher) on every node, a stress test such as gpu-burn run for a stated number of hours, and a storage throughput benchmark at a stated block size and concurrency.
2. The pass thresholds, computed the way sections 1 through 5 compute them, not asserted: the theoretical bisection bandwidth for the actual delivered port count and speed, a busbw target near 90 percent of that ceiling, zero fatal Xid events over the burn-in window, and a sustained storage throughput figure in GB/s per GPU.
3. The duration, stated in hours. A fabric or a GPU can pass a five-minute test and fail an eight-hour one; a burn-in window measured in hours, not minutes, is what catches thermal throttling and intermittent hardware faults.
4. Who runs it and who watches, named. Raw logs reviewed by, or a test executed in the presence of, the buyer’s own technical representative, not a summary PDF the vendor produces unsupervised.
5. A holdback tied to the result, not the install date. A common contracting structure withholds a defined minority of the total contract value until the acceptance test passes, with a stated cure period and re-test procedure if it does not. The exact percentage and cure period are a negotiation, not a fact this post can hand you, but the principle is fixed: the racks going into the room is a delivery milestone, not a payment milestone.

The order to check things in

1. Compute the theoretical bisection bandwidth yourself from the delivered port count and port speed, and demand a measured busbw around 90 percent of it.
2. Ask for the measured busbw number from an all_reduce_perf run, not the NIC’s rated link speed. They are different numbers by design.
3. Ask for the leaf-to-spine port ratio in writing. A switch with the right port count can still be cabled at 2:1 or 3:1.
4. Ask for a DCGM run-level 3+ diagnostic and an hours-long burn-in log across every GPU, not a sample, and check it for Xid 79, 63, and 48 events.
5. Ask for a measured storage throughput number in GB/s per GPU, sustained, not the usable capacity in petabytes.
6. Put all five of the above into the contract as the acceptance gate, with a defined holdback and a re-test path, so the final payment depends on the cluster that was measured, not the cluster that was promised.

None of this requires becoming an infrastructure engineer. It requires knowing which six numbers exist, how each is computed from information the vendor already has, and refusing to release the balance of the payment until someone hands you the measured versions of all six. For the arithmetic behind what that same cluster costs to run once it passes these tests, see what it actually costs to run a GPU cluster.