SC22 It is protected to say that liquid cooling was a sizzling subject on the Supercomputing convention in Dallas this week.
So far as the attention may see, the showroom was filled with liquid-cooled servers, oil-filled immersion cooling tanks, and all of the equipment, pumps, and coolant distribution items (CDUs) you could possibly have to implement the know-how in an information middle. .
Since this can be a convention on excessive efficiency computing, the emphasis on thermal administration mustn’t come as a shock. However with 400W CPUs and 700W GPUs now out there, it isn’t simply an HPC or AI downside. As extra firms look so as to add AI/ML-enabled programs to their knowledge facilities, 3kW, 5kW, and even 10kW programs aren’t so loopy anymore.
So here is a breakdown of the liquid cooling package that caught our eye at this yr’s present.
Direct liquid cooling
The overwhelming majority of liquid cooling programs proven at SC22 are of the direct liquid selection. They commerce copper or aluminum warmth sinks and followers for chilly plates, rubber tubing, and equipment.
If we’re being trustworthy, these chilly dishes look just about the identical. Principally they’re only a hollowed out block of metallic with an inlet and outlet for the fluid to undergo. Notice that we’re utilizing the phrase “liquid” right here as a result of liquid-cooled programs can use any variety of refrigerants that aren’t essentially water.
A Supermicro liquid-cooled server geared up with CoolIT chilly plates. – Click on to enlarge
In lots of instances, OEMs supply their chilly plates from the identical distributors. For instance, CoolIT offers liquid cooling {hardware} for varied OEMs, together with HPE and Supermicro.
Nevertheless, that is to not say there is not a chance for differentiation. The inside of those chilly plates is full of microfin arrays that may be adjusted to optimize fluid circulation by way of them. Relying on the scale or the variety of dies to be cooled, the inside of those chilly plates can differ fairly a bit.
A lot of the liquid-cooled programs we noticed on the present flooring used some sort of rubber tubing to attach the chilly plates. Because of this the liquid solely cools particular elements just like the CPU and GPU. So whereas the majority of the followers may be eliminated, some airflow continues to be required.
HPE demonstrates its newest liquid-cooled Cray EX blades utilizing AMD’s 96-core Epyc 4 CPUs. – Click on to enlarge
The Lenovo Neptune and HPE Cray EX blades have been the exception to this rule. Their programs are designed particularly for liquid cooling and are filled with copper tubing, bus blocks, and chilly plates for all the pieces together with CPUs, GPUs, reminiscence, and NICs.
With this method, HPE has succeeded in packing eight AMD 400W Epyc 4 Genoa CPUs right into a 19-inch chassis.
A liquid-cooled Lenovo Neptune server configured with two AMD Genoa CPUs and 4 Nvidia H100 GPUs. – Click on to enlarge
Lenovo, in the meantime, confirmed off a 1U Neptune system designed to chill a pair of 96-core Epycs and 4 of Nvidia’s H100 SXM GPUs. Relying on the implementation, producers declare that their direct liquid cooling programs can take away between 80 and 97 % of the warmth generated by the server.
immersion cooling
One of many extra unique liquid cooling applied sciences on show at SC22 was immersion cooling, which has made a comeback in recent times. These programs can seize one hundred pc of the warmth generated by the system.
As a substitute of retrofitting the server with chilly plates, immersion cooling tanks, like this one from Submer, submerge them in a non-conductive liquid. click on to enlarge
As loopy because it sounds, we have been soaking laptop elements in non-conductive fluids to maintain them cool for many years. One of the well-known programs for utilizing immersion cooling was the Cray 2 supercomputer.
Whereas the fluids utilized in these programs differ from provider to provider, artificial oils from Exxon or Castrol or specialty coolants from 3M should not unusual.
Submer was one in every of a number of immersion cooling firms to point out off their know-how at SC22 this week. The corporate’s SmartPods look a bit such as you fill a chest freezer filled with oil and begin inserting servers vertically from the highest.
Submer gives tanks in varied sizes which might be roughly equal to conventional half and full racks. These tanks are rated for 50-100kW of warmth dissipation, placing them on par with rack-mounted air and liquid cooling infrastructure when it comes to energy density.
Submer’s tank helps OCP OpenRack type elements like these tri-bladed Intel Xeon programs – Click on to Broaden
The demo tank had three 21-inch servers, every with three dual-socket Intel Sapphire Rapids blades, in addition to a normal 2U AMD system that had been transformed to be used of their tanks.
Nevertheless, we’re instructed that the variety of mods required, particularly to the OCP chassis, is fairly negligible, with the one actual modifications being altering the transferring components of issues like energy provides.
Unsurprisingly, immersion cooling complicates upkeep and is a little more concerned than air or direct liquid cooling.
Iceotope’s twist on immersion cooling makes use of the server chassis as a reservoir. – Click on to enlarge
Not all present flooring immersion cooling setups require gallons of specialised fluids. Iceotope’s in-chassis immersion cooling system was one instance. The corporate’s sealed server chassis capabilities like a reservoir with the motherboard submerged in a couple of millimeters of liquid.
A redundant pump on the rear of the server recirculates oil to hotspots such because the CPU, GPUs, and reminiscence earlier than passing the new fluids by way of a warmth exchanger. There, the warmth is transferred to a facility water system or rack-scale Coolant Distribution Models (CDUs).
Help infrastructure
No matter whether or not you employ direct-chip or immersion cooling, each programs require extra infrastructure to extract and dissipate warmth. For direct liquid-cooled configurations, this may embrace distribution manifolds, rack-level plumbing, and most significantly, a number of CDUs.
Massive rack dimension CDUs can be utilized to chill a whole row of server cupboards. For instance, Cooltera confirmed a number of massive CDUs able to offering as much as 600 kW of cooling to an information middle. Or, for smaller deployments, a rack-mounted CDU can be utilized. We checked out two examples from Supermicro and Cooltera, which offered between 80 and 100 kW of cooling capability.
A Cooltera Rack Mounted Refrigerant Distribution Unit – Click on to Broaden
These CDUs are made up of three primary elements: a warmth exchanger, redundant pumps to flow into coolant by way of the racks, and a filtration system to forestall particles from clogging essential elements like chilly plate microfins.
The best way wherein warmth is definitely faraway from the refrigeration system largely will depend on the kind of warmth exchanger that’s used. Liquid to air warmth exchangers are among the many easiest as a result of they require the fewest modifications to the set up itself. The Cooltera CDU proven right here makes use of massive radiators to discharge warmth captured by fluid within the knowledge middle’s sizzling aisle.
Along with the pumps and filtration, this Cooltera CDU options an built-in liquid to air warmth exchanger. – Click on to enlarge
Nevertheless, a lot of the CDUs we noticed at SC22 used liquid-to-liquid warmth exchangers. The concept right here is to make use of a separate facility-wide water system to maneuver the warmth collected from a number of CDUs to dry coolers on the surface of the constructing the place it’s dissipated to the air. Or, as an alternative of dumping warmth into the environment, some knowledge facilities, like Microsoft’s newest facility in Helsinki, have linked their facility’s water programs to district heating programs.
The state of affairs is way the identical for immersion cooling, though lots of the CDU elements, reminiscent of pumps, liquid-to-liquid warmth exchangers, and filtration programs, are constructed into the tanks. All that’s actually required is that they be linked to the ability’s water system.
Liquid cooling adoption on the rise
Whereas liquid cooling represents solely a fraction of knowledge middle thermal administration spend immediately, hotter elements and better rack energy densities are starting to drive adoption of the know-how.
Based on a current Dell’Oro Group report, spending on immersion and liquid cooling gear is anticipated to succeed in $1.1 billion or 19% of thermal administration spending by 2026.
In the meantime, rising vitality costs and a better emphasis on sustainability make liquid cooling enticing on different ranges. Placing apart the practicality of cooling a 3kW server with air, 30-40 % of an information middle’s vitality consumption may be attributed to air-con and air dealing with gear required to maintain programs at working temperature. .
So whereas server distributors have discovered methods to air-cool servers as much as 10kW, within the case of Nvidia’s DGX H100, at these thermal and energy densities, there are exterior incentives to cut back the ability consumption now getting used for computing. ®
– Liquid and immersion is the new cool at Supercomputing ‘22 • The Register
