Liquid and immersion is the new cool at Supercomputing '22(theregister.com)
theregister.com
Liquid and immersion is the new cool at Supercomputing '22
https://www.theregister.com/2022/11/19/liquid_cooling_sc22/
33 comments
I think this is a nice summary. As a point for water cooling, it's easier to use the generated heat; it can for instance be used to heat nearby buildings. With air cooling this is much more difficult. I think watercooling will become dominant in new data centers as environmental laws sets stricter requirements for heat waste.
> With air cooling this is much more difficult
To my understanding, DCs use air conditioning systems for cooling. These are already heat pumps - why would it be more difficult to just pump that heat into district heating instead of outside air?
To my understanding, DCs use air conditioning systems for cooling. These are already heat pumps - why would it be more difficult to just pump that heat into district heating instead of outside air?
Because the temperature spread a given heat pump uses isn't really suitable for district heating, your output water would only be in the 50 to 60°C range at most, too low for typical district heating systems. One can get around that with multi-stage heat pumps or different coolants, but both are expensive investments typically not viable for existing systems.
If it's not enough for district heating because of the distance and heat losses, it could be enough for a greenhouse right next to a DC.
Economically the greenhouse would have a distinct advantage of using free heat, and already having enough electricity wired nearby.
This, of course, only makes sense during cold seasons, or somewhere like Alaska or Sweden. During hot months, 40-50°C of heat seem completely useless.
Maybe next to a sea such waste heat could help evaporatively desalinate seawater, while cooling back to reusable temperatures.
Economically the greenhouse would have a distinct advantage of using free heat, and already having enough electricity wired nearby.
This, of course, only makes sense during cold seasons, or somewhere like Alaska or Sweden. During hot months, 40-50°C of heat seem completely useless.
Maybe next to a sea such waste heat could help evaporatively desalinate seawater, while cooling back to reusable temperatures.
Yes, heat reuse is easier with water cooling since the losses are smaller, but even air cooling will produce warm water in the end. There will be coming regulations requiring a certain amount of heat reuse, but in a first step, this probably won't force direct water cooling.
For those regulations, there is also the important aspect of free cooling, i.e just using outside air as a medium or using an air-air heat exchanger like a Tokyo Wheel. Mandated heat reuse (without appropriate exceptions) would make those illegal, however in certain situations those are the best (environmentally and financially) methods of cooling. E.g. in a moderately hot summer you can use 20-30°C outside air without a problem, but if you would mandate heat reuse at that time, you would have lots and lots of excess heat on your hands with nobody buying it. So while some regulation will be coming, I hope and think that it wouldn't really issue a strict mandate to reuse heat and allow free air cooling, which would exclude direct water cooling.
For those regulations, there is also the important aspect of free cooling, i.e just using outside air as a medium or using an air-air heat exchanger like a Tokyo Wheel. Mandated heat reuse (without appropriate exceptions) would make those illegal, however in certain situations those are the best (environmentally and financially) methods of cooling. E.g. in a moderately hot summer you can use 20-30°C outside air without a problem, but if you would mandate heat reuse at that time, you would have lots and lots of excess heat on your hands with nobody buying it. So while some regulation will be coming, I hope and think that it wouldn't really issue a strict mandate to reuse heat and allow free air cooling, which would exclude direct water cooling.
>
>so you will always fare better (and usually more environment-friendly) with air cooling.
Sorry but this is not true. One of the key drivers of liquid cooling IS efficiency. Liquids are for more effective at heat transfer than air. Additionally, a liquid can carry a lot more heat. Yes, you can run higher density, you can also indirectly free cool servers year-round, even in tropical climates.
>so you will always fare better (and usually more environment-friendly) with air cooling.
Sorry but this is not true. One of the key drivers of liquid cooling IS efficiency. Liquids are for more effective at heat transfer than air. Additionally, a liquid can carry a lot more heat. Yes, you can run higher density, you can also indirectly free cool servers year-round, even in tropical climates.
You are both right and wrong, because you (and I) are not clearly distinguishing between different kinds of efficiency and effectiveness.
Liquid cooling is, without question, more /effective/. It makes things cooler, it has a far higher capacity for heat transport.
However, with /efficiency/, you do weigh the results against the effort put in: A solution is more efficient if the quotient result/effort is higher. However, in case of cooling, there are two ways to measure effort: energy expended or money expended. In the case of energy efficiency, yes, liquid cooling is better, because of the aforementioned higher effectiveness of heat transport, lesser energy expenses for pumps/fans and easier energy reuse. However, financially, all that liquid handling, custom piping, pumping, coolant, etc. is by far more expensive than a few air ducts and fans. Even with air-air heat pumps, air cooling is still less expensive, so better financial efficiency. That's why everybody is doing it as a default atm.
Liquid cooling is, without question, more /effective/. It makes things cooler, it has a far higher capacity for heat transport.
However, with /efficiency/, you do weigh the results against the effort put in: A solution is more efficient if the quotient result/effort is higher. However, in case of cooling, there are two ways to measure effort: energy expended or money expended. In the case of energy efficiency, yes, liquid cooling is better, because of the aforementioned higher effectiveness of heat transport, lesser energy expenses for pumps/fans and easier energy reuse. However, financially, all that liquid handling, custom piping, pumping, coolant, etc. is by far more expensive than a few air ducts and fans. Even with air-air heat pumps, air cooling is still less expensive, so better financial efficiency. That's why everybody is doing it as a default atm.
The problem with liquid cooling, ultimately, is you're still dumping to air. You're actually doing not much more with liquid cooling than just changing where the heat concentration is located.
Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time.
Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time.
>The problem with liquid cooling, ultimately, is you're still dumping to air.
There's a lot of things you can do outside of a server chassis that you cannot do inside of a chassis. You can have heat exchangers with far more surface area than what you are volume constrained to inside of a chassis. You can run evaporative cooling towers. If you're in the right climate, you can even radiate that heat to space.
>You're actually doing not much more with liquid cooling than just changing where the heat concentration is located.
You're using a more efficient heat transfer medium which does a lot of things for you. You can downsize your chiller tonnage, reduce/eliminate CRAC/CRAH units, and all the UPS required to keep those systems running. All of this translates to CAPEX and OPEX savings.
>Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time.
No one is suggesting that you can plonk some servers into a tank of liquid and the thermal energy magically disappears. Of course, there has to be a thermodynamic "sink" that you are dumping to... the atmosphere, the ocean, the ground, space, etc. Liquid cooling is more efficient means of transferring heat from source to sink.
There's a lot of things you can do outside of a server chassis that you cannot do inside of a chassis. You can have heat exchangers with far more surface area than what you are volume constrained to inside of a chassis. You can run evaporative cooling towers. If you're in the right climate, you can even radiate that heat to space.
>You're actually doing not much more with liquid cooling than just changing where the heat concentration is located.
You're using a more efficient heat transfer medium which does a lot of things for you. You can downsize your chiller tonnage, reduce/eliminate CRAC/CRAH units, and all the UPS required to keep those systems running. All of this translates to CAPEX and OPEX savings.
>Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time.
No one is suggesting that you can plonk some servers into a tank of liquid and the thermal energy magically disappears. Of course, there has to be a thermodynamic "sink" that you are dumping to... the atmosphere, the ocean, the ground, space, etc. Liquid cooling is more efficient means of transferring heat from source to sink.
Isn’t that the point?
You use a liquid to transport the heat from denser compute (lowering costs), then run it through a high efficiency radiator (eg, drip/spray towers like with industrial AC). Your compute is denser and your interface with the air is more efficient.
You use a liquid to transport the heat from denser compute (lowering costs), then run it through a high efficiency radiator (eg, drip/spray towers like with industrial AC). Your compute is denser and your interface with the air is more efficient.
Yes. Liquid cooling is surface area arbitrage. You turn small surface area into large area with a good conductor between them
What's your opinion of the Microsoft Ignite 2022 demo by Mark Russinovich of their next-gen immersion-cooled setup for Azure compute nodes? Link: https://youtu.be/PO5ijv6WDv0?t=370
Not good. As you can see, even the upper side connections are submerged, and plugs still look like standard components. Nothing to say of the backplane connectors on the lower side and RAM and CPU connections. They seem to just have taken a blade enclosure and flipped it backside up. So yes, it looks expensive, maybe impressive for some audience, but totally impractical even though they claim it to be a custom design.
Your main concern seems to be with ongoing maintenance, but my understanding is that the "hyperscalers" like Azure don't do maintenance. They assemble everything once and then let failed components remain in-place and powered off.
Google championed this over a decade ago. They figured that the cost of staff, spare parts, diagnostics, etc... outweighed the cost of the hardware. If you factor in the complexity of liquid cooling, it might not make sense to repair hardware even a single compute node costs north of $20K.
Google championed this over a decade ago. They figured that the cost of staff, spare parts, diagnostics, etc... outweighed the cost of the hardware. If you factor in the complexity of liquid cooling, it might not make sense to repair hardware even a single compute node costs north of $20K.
We had 8 systems in the Top 500 list at Weta Digital back in 2008 which were liquid cooled HP servers for rendering Avatar. We used one of the techniques mentioned in the article: liquid to liquid heat exchangers. Each rack was fully sealed with its own heat exchanger which pumped in cool water and sent off warm water to chillers on the roof. Actually the warm water condensing would attract seagulls in the winter which was a problem since we were right next to the airport. The datacentre was designed by an engineering firm which usually built dairy factories. Every once in a while something would go wrong with the cold water delivery system and the servers would overheat and shut down in about 30 seconds.
A couple of interesting factors leading to liquid cooling in the commercial market (always been prominent in the top 500) and why I think this time it’s different.
First, we are designing smaller transistor leading to leakage and thus increasing chip thermal design power (TDP). We are also using bigger chips and chiplets also increasing TDP.
TDP on a GPU went from 200w to 1 kW in the past 5 years and even CPU are now close to 500w. At 25k a pop avoiding throttling due to heat restriction is a must and is becoming incredibly hard with air cooling as air is an amazing insulator.
Second, networking is extremely expensive with the cost of top-end InfiniBand cable close to 1000$ a foot. This means you want to cram your CPU /GPU as close as possible to keep the connection electrical if possible and minimize optical cable lengths. This also decreases latency and increase cluster performance. On a 50 000 cable deployment, the saving can be … significant.
Third, we are now seeing the emergence of foundational model that requires government lab levels of interconnected compute. This is a new paradigm in the private sector and consume power in the 20-30 MW range. An increase in efficiency of 10% (from 1.2 to 1.1 PUE) brings huge saving on the power cost side plus important environmental benefits. It also becomes much easier to recuperate heat.
That’s why we’re seeing major announcement at OCP i.e. meta going the liquid cooling way.
I personally believe that’s where things are going and we are building a facility to optimize for the above points: https://www.qscale.com/
First, we are designing smaller transistor leading to leakage and thus increasing chip thermal design power (TDP). We are also using bigger chips and chiplets also increasing TDP.
TDP on a GPU went from 200w to 1 kW in the past 5 years and even CPU are now close to 500w. At 25k a pop avoiding throttling due to heat restriction is a must and is becoming incredibly hard with air cooling as air is an amazing insulator.
Second, networking is extremely expensive with the cost of top-end InfiniBand cable close to 1000$ a foot. This means you want to cram your CPU /GPU as close as possible to keep the connection electrical if possible and minimize optical cable lengths. This also decreases latency and increase cluster performance. On a 50 000 cable deployment, the saving can be … significant.
Third, we are now seeing the emergence of foundational model that requires government lab levels of interconnected compute. This is a new paradigm in the private sector and consume power in the 20-30 MW range. An increase in efficiency of 10% (from 1.2 to 1.1 PUE) brings huge saving on the power cost side plus important environmental benefits. It also becomes much easier to recuperate heat.
That’s why we’re seeing major announcement at OCP i.e. meta going the liquid cooling way.
I personally believe that’s where things are going and we are building a facility to optimize for the above points: https://www.qscale.com/
> InfiniBand cable close to $1000 a foot
Wait... wat? I thought InfiniBand just used fibre!
Wait... wat? I thought InfiniBand just used fibre!
https://www.naddod.com/products/18224.html that's the 2018 200g standard at over 1000$ a meter. Can't find public pricing but the new 400/800 gb/s are more expensive!
That price is 95% the endpoints, 5% the cable. The fibre in between the QSFP adapters on the end is practically free.
The 30m pricing is almost the same: https://www.naddod.com/products/18231.html
The 30m pricing is almost the same: https://www.naddod.com/products/18231.html
1m $1,087
10m $1,124
30m $1,196
It's absolutely delightful to see concepts come many full circles like this. I remember the Cray-2 cooling towers as the most futuristic computer in the world even in the 90s, and it inspired me to sink my old system in mineral oil in the early 00s. I did the research and found the stuff in the Cray towers (Fluorinert) was $many/gallon, and the smallest volume they sold it was something like 55gal drums. Mineral oil was $cheap enough, also inert, and so I got a 5 gallon tub of the stuff. I built a plexiglass case, sealed it with silicone caulk, overclocked the CPU, hooked up an industrial-strength aquarium circulation pump and... rapidly went nowhere with a disgusting, oily motherboard. And a ridiculously heavy case filled with, essentially, hazmat.
It was a terrible idea back then and still seems bad today. I'll be interested to see what the next cycle brings, if it's back to spot cooling or if immersion really is the way of the future.
It was a terrible idea back then and still seems bad today. I'll be interested to see what the next cycle brings, if it's back to spot cooling or if immersion really is the way of the future.
I run large scale data centers. Crypto mining has been pushing a lot of this technology ahead as well [1] (yea, yea).
Liquid cooling is nice on paper, but as others have noted, it is messy AF.
Another huge issue that you don't normally think aobut is staffing... some data centers are in the middle of nowhere. It is hard enough to just fine skilled enough labor to work reliably... getting them trained up to do maintenance on wet systems is a whole another level of pain.
It is great for small systems... not so good at scale. It is amazing how proper airflow with fans can solve most cooling needs. Even if the outside temps are high as well.
[1] https://twitter.com/SenTedCruz/status/1564441153306406912
Liquid cooling is nice on paper, but as others have noted, it is messy AF.
Another huge issue that you don't normally think aobut is staffing... some data centers are in the middle of nowhere. It is hard enough to just fine skilled enough labor to work reliably... getting them trained up to do maintenance on wet systems is a whole another level of pain.
It is great for small systems... not so good at scale. It is amazing how proper airflow with fans can solve most cooling needs. Even if the outside temps are high as well.
[1] https://twitter.com/SenTedCruz/status/1564441153306406912
Let's be honest here..."...getting them trained up to do..."...probably doesn't exist these days. In my 3 decades of HR interviewing exits, near 19 out of 20 replied no training...heck some don't get any proper intro by their manager in charge. During interviewing, hiring managers already expect candidates are better than the one leaving complete with company internal workflow knowledge. During 2008, the expectation of resume thickness exceed a doctorate-master thesis.
Last time I looked into this the best available coolant was the 3M Novec line, but it was around $400 usd/gallon. Is it available anywhere else for less?
And I could be wrong and this could be from a previous iteration of this “inert” liquid dialectic insulators chemical line from 3M but I believe some of them have at least in the past had a less the ideal toxicity profile or have had issues with the normal and inevitable slow chemical breakdown from ionising and cosmic radiation… producing notably toxic gasses you definitely wouldn’t want slowly building up unmonitored in an enclosed space for the years.
Don’t get me wrong, the Novec line and it’s ancestors and siblings like the stuff that cooled the old Cray supercomputers… are the sort of Chemical wizardry that had chemical engineering on the prospective career list when I was in high school. I just remember seriously investigating using them for a project regardless of the price and coming across some sort of chemical safety red flag. So if anyone knows/recognises what I’m half remembering, feel free to share the details!
Don’t get me wrong, the Novec line and it’s ancestors and siblings like the stuff that cooled the old Cray supercomputers… are the sort of Chemical wizardry that had chemical engineering on the prospective career list when I was in high school. I just remember seriously investigating using them for a project regardless of the price and coming across some sort of chemical safety red flag. So if anyone knows/recognises what I’m half remembering, feel free to share the details!
We used this stuff for a small pilot project: https://www.engineeredfluids.com/electrocool
Pre-covid we paid roughly $225 for 20 liters, not bulk pricing. Not quite apples to apples as 3M Novec is a 2 phase coolant, we went with single phase.
Pre-covid we paid roughly $225 for 20 liters, not bulk pricing. Not quite apples to apples as 3M Novec is a 2 phase coolant, we went with single phase.
[deleted]
Not seeing anything mentioning 3m's improved Fluorinert phase change dielectric coolants?
Fortunately I had the pleasure of using these coolants in high school when I was hired to help build a commercial scale ASIC farm in the midwest - we used about 200 gallons of 3m Novec 7000. Which is insane since that coolant cost around $180/gal even at a steep discount ($36,000 in total). And that's after the shipment arrived with 10% of the bottles "empty" since the container sat in a hot port - coolant simply slipped past gaskets on the glass bottles.
Fortunately I had the pleasure of using these coolants in high school when I was hired to help build a commercial scale ASIC farm in the midwest - we used about 200 gallons of 3m Novec 7000. Which is insane since that coolant cost around $180/gal even at a steep discount ($36,000 in total). And that's after the shipment arrived with 10% of the bottles "empty" since the container sat in a hot port - coolant simply slipped past gaskets on the glass bottles.
I want to try immersing a RPI in mineral oil. It should fit in a metal pint paint can; epoxy around the ports and it should at least only leak slowly when tipped over.
That should work, but I think for bulky cooling I'd just stick the pi to a half inch thick square of aluminum.
100%. It has to be that easy. I hope soon.
do it!
Wasn't the Cray 2 liquid immersion cooled, 35 years ago, famously?
https://en.wikipedia.org/wiki/Cray-2
https://en.wikipedia.org/wiki/Cray-2
But imho especially immersion cooling is a dead end. Contact problems with cabling lower reliability because the isolating coolant creeps between the metal surfaces. Maintenance is unbelievably convoluted, not only do you have to drain all the fluid out to pull a machine, you will need to clean each and every plug, socket and pin you touch because the contact surfaces will be coated with isolating coolant. This could only be fixed with a complete redesign of everything, so that all or at least most contacts are "dry". Using fluid-tight plugs is very expensive, and anyways, most ideas there won't work for important components such as CPU and RAM.
Liquid cooling will fare better, because you can strike a balance between high-power components like CPU and GPU where you slap a liquid cooler on. And low-power components like most of the board you just cool by air as usual. But even there, maintenance is somewhat of a hassle, in addition to all the cabling you also need to deal with piping, leaks, drippage (which aren't supposed to happen, but trust me, they do). I think the only field where this is really going to be used is HPC for applications where floorspace is limited. E.g. if your network cabling must be within a certain length limit of your central network node, so you need to cram all your racks for the supercomputer you are building within some 20m circle or the like.
Less top-of-the-line HPC and datacenters are better served by just limiting power density and continuing air cooling as usual. Datacentre floorspace is expensive, but not _that_ expensive, so you will always fare better (and usually more environment-friendly) with air cooling.