How to burn in a GPU cluster before you accept it

A GPU cluster can pass every acceptance test on delivery day and still fall over in week two. The tests you run at handoff are a snapshot: they tell you the fabric hits its bandwidth number and the GPUs enumerate correctly right now. They do not tell you which GPU will start throwing uncorrectable memory errors after six hours at full power, which optical link will flap the first time a real training job saturates it, or which memory module is marginal enough to fail under a week of load but fine for a five-minute check. Burn-in is how you find those faults before you have paid for them.

Burn-in is not the same as acceptance testing

Acceptance testing asks “does this cluster hit its numbers right now”: bisection bandwidth, collective-operation throughput, storage speed, the health of the worst GPU in the fleet. We wrote those gates up separately in how to tell if a GPU cluster actually works. Burn-in asks a different question: “does this cluster stay healthy under days of sustained stress.” You want both, and they are not interchangeable. A snapshot catches a cluster that was built wrong. A soak catches a cluster that was built with marginal parts, which is the more common and more expensive failure, because it surfaces after the money has already moved. Burn-in also leaves you with a clean performance baseline to measure against later, so when a job slows down in month three you can tell whether the hardware drifted or your code did.

The shape of a real burn-in: two tiers

A credible burn-in runs in two stages, and the second does not begin until the first is clean.

Per-host first. Each node is stressed on its own: CPU, memory, local storage, and every GPU driven to sustained full load for roughly a day, long enough that thermal and power faults have time to appear. A node that fails here never makes it into the fabric test.

Cluster-wide second. Once every node is individually clean, the whole cluster is soaked together: the GPU interconnect, collective operations across nodes, and the network fabric under sustained load for several days. This is the tier that catches the faults that only show up at scale, a link that flaps under a full all-reduce, a rail that was miscabled, a switch that drops under real traffic.

Across both tiers, the cluster is monitored continuously, not just measured at the start and the end. Error counters, thermal and power throttling, and node liveness are watched the whole way through, because a fault that appears on hour 40 of a multi-day soak is exactly the kind you are running the soak to find. This two-tier, continuously-monitored structure is simply how a burn-in worth trusting is run, and it is the shape we look for whenever we vet a cluster on a buyer’s behalf.

What a soak catches that a snapshot cannot

The whole point of burn-in is time and heat. The faults it surfaces are the ones that are invisible in a quick check:

• GPUs that are fine cold and degrade hot: uncorrectable memory errors, memory that has to be remapped, hardware faults that only trip after sustained load.
• Memory and CPU that pass a short test but fault under a long one.
• Marginal optics and cabling: fabric links that stay up while idle but flap or drop under a real collective.
• Throttling: clocks and power that hold for five minutes but sag under an hour at full tilt, quietly capping the performance you paid for.
• Drift: a cluster that is measurably slower at the end of the soak than at the start, which tells you something is already on its way out.

Every one of these is cheap to make the supplier fix during burn-in and expensive to discover after acceptance, once the remediation obligation has quietly moved to you.

Not every finding is a stop-ship

Burn-in is not pass-or-scrap. The useful frame is to split findings into hard-fail criteria and scored criteria. A hard-fail criterion is one a node must pass: uncorrectable memory errors, failed GPUs, dropped fabric links. A scored criterion is one where a miss is allowed only with a documented disposition: remediated now, waived on the record with a reason, or scheduled with a date. That distinction is what keeps a burn-in from being either theater, where everything is waved through, or paralysis, where one marginal reading blocks an otherwise working cluster. You want the hard line to be genuinely hard, and everything else written down rather than argued verbally.

The part that protects your money

Here is the commercial reason this matters, and it is the same theme as the rest of these posts: your leverage is highest before you sign. Two things belong in the deal before you release payment.

First, proof of a clean burn-in, or the time and access to run your own. If a supplier cannot show you a burn-in result and will not give you the cluster long enough to run one, that is itself the answer to your question. A supplier who runs a real burn-in already has the artifacts, and producing them costs nothing.

Second, remediation ownership pinned to the timeline. During burn-in, fixing a bad GPU or a flapping link is the supplier’s job and the supplier’s cost. After you countersign acceptance, it becomes yours. That single line, who fixes what and until when, is worth more than a small movement on the hourly rate, because three marginal nodes you inherited will cost you far more in lost training time than you saved at the table.

Acceptance itself should be a document, not a handshake: a per-node inventory matched against the purchase order, a pinned firmware and driver baseline recorded as an artifact, all hard criteria passing, every scored finding dispositioned, and your countersignature. That certificate is what you point to in month four when a node misbehaves and the conversation turns to whose problem it is.

What to demand, in one list

1. Proof of a successful burn-in, or the right and the time to run one, before acceptance and before payment.
2. A two-tier soak: every node individually clean first, then the whole fabric under sustained load for several days.
3. Continuous monitoring across the whole soak, not just a reading at the start and the end.
4. A hard-fail versus scored split, with every scored miss written down and dispositioned rather than waved through.
5. A pinned firmware and driver baseline recorded per node, so you know exactly what you accepted.
6. A signed acceptance certificate, with remediation the supplier’s responsibility for the duration of the burn-in.

None of this requires you to run the tests yourself. It requires you to make burn-in a term of the deal instead of a favor you hope the supplier did. The cluster that passed a five-minute check and the cluster that survived a five-day soak are not the same purchase, and only one of them is the one worth paying for.