Liquid cooling next-gen servers getting hands-on(servethehome.com)
servethehome.com
Liquid cooling next-gen servers getting hands-on
https://www.servethehome.com/liquid-cooling-next-gen-servers-getting-hands-on-3-options-supermicro/
57 comments
>The massive problem here is that its really difficult to hotswap anything.
And this problem has been there since Day 1 ( For well over a decade ). And no one seems to have solve it. It make sense if you treat the whole Rack as a single entity with water cooling. But for individual server I still dont understand how and why liquid cooling could benefits.
And this problem has been there since Day 1 ( For well over a decade ). And no one seems to have solve it. It make sense if you treat the whole Rack as a single entity with water cooling. But for individual server I still dont understand how and why liquid cooling could benefits.
The problems that FAANG have aren't the problems most of us have. Edit: That don't work for those companies. Different levels of scale certainly.
https://xkcd.com/1737/
https://xkcd.com/1737/
>> I still dont understand how and why liquid cooling could benefits.
Because liquid cooling is only a marginal benefit for current hardware, hardware designed to be air cooled. If liquid cooling becomes a norm we should start seeing hardware designed with the expectation of liquid cooling. That could mean greater component density and greater power concentration --> greater speeds. One wonders what CPUs would be like if they were designed with the expectation of liquid cooling, water within a millimetre from the heat source rather than the metal heat spreaders that currently top most chips.
How many fans are in a modern server rack? Liquid cooling could theoretically replace all of those with one moving part, one pump impeller.
Because liquid cooling is only a marginal benefit for current hardware, hardware designed to be air cooled. If liquid cooling becomes a norm we should start seeing hardware designed with the expectation of liquid cooling. That could mean greater component density and greater power concentration --> greater speeds. One wonders what CPUs would be like if they were designed with the expectation of liquid cooling, water within a millimetre from the heat source rather than the metal heat spreaders that currently top most chips.
How many fans are in a modern server rack? Liquid cooling could theoretically replace all of those with one moving part, one pump impeller.
A few years back IBM had a test line of liquid cooled modular components. Lovely things had everything connected & sealed and you could stack multiple identical components. Didn't go anywhere at the time but, the idea could be modified and re-implemented these days similar to your suggestions with great benefits.
> How many fans are in a modern server rack? Liquid cooling could theoretically replace all of those with one moving part, one pump impeller.
Or maybe more than one impeller, because at least with the fans, cooling continues if there's a failure in one of them.
Or maybe more than one impeller, because at least with the fans, cooling continues if there's a failure in one of them.
Well, perhaps coolant would be delivered to racks at a particular psi/volume just as electricity is today provided at a particular voltage and current. Then an entire datacenter would be served by a handful of massive pumps. Server racks might then have no moving parts, which couldn't harm reliability.
>I doubt its going to be mainstream anytime soon.
I would like to have a quiet and powerful machine for deep learning. The best one I found come from NVIDIA DGX station. It uses liquid cooling and is <35dB acoustics. The problem is that it is too expensive and, last I check, I think it requires buying support contract from third-party.
If liquid cooling is what it takes to have a quiet powerful machine, I hope it will be mainstream.
Source: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...
I would like to have a quiet and powerful machine for deep learning. The best one I found come from NVIDIA DGX station. It uses liquid cooling and is <35dB acoustics. The problem is that it is too expensive and, last I check, I think it requires buying support contract from third-party.
If liquid cooling is what it takes to have a quiet powerful machine, I hope it will be mainstream.
Source: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...
What we did in the VFX world is just use hardware VDI.
Shove in a https://www.teradici.com/ card, and bam, low latency, high colour accuracy, high resolution remote video.
This allowed us to use the huge machines that drove realtime VFX (your flames, baselite and editing rigs) in utter silence.
Shove in a https://www.teradici.com/ card, and bam, low latency, high colour accuracy, high resolution remote video.
This allowed us to use the huge machines that drove realtime VFX (your flames, baselite and editing rigs) in utter silence.
Enthusiast water cooling would work for you, which if you're not comfortable with yourself you can get from people like Maingear.
But unless you need 4 GPUs, even just 2 GPUs with normal modern air cooling would still be reasonably quiet.
For the overwhelming majority of consumer-oriented products, you don't need water cooling to be silent (or near-silent). Those tower coolers and triple-slot GPU coolers are really good at getting rid of the heat in a noise-efficient manner.
But unless you need 4 GPUs, even just 2 GPUs with normal modern air cooling would still be reasonably quiet.
For the overwhelming majority of consumer-oriented products, you don't need water cooling to be silent (or near-silent). Those tower coolers and triple-slot GPU coolers are really good at getting rid of the heat in a noise-efficient manner.
Why does it need to be quiet though? Don't most of these live in a datacentre with no humans around? I assume you will say you want to keep it in your office or similar, but then I'll just ask again - why exactly?
Its designed to be in the office, so having a default sound level of 75dB isn't going to be a winner.
Plus I suspect that having watercooling for that machine allows it to fit in a much smaller space. We had a machine that contained 4 RTX 2080s (or maybe the generation before) and it took up a huge amount of space, and never went at full speed, because of the cooling (those consumer cards are not designed for density.)
Plus I suspect that having watercooling for that machine allows it to fit in a much smaller space. We had a machine that contained 4 RTX 2080s (or maybe the generation before) and it took up a huge amount of space, and never went at full speed, because of the cooling (those consumer cards are not designed for density.)
The DGX Station designed to run on 35dBA. So Nvidia thinks it need to be run silently because of it should be a monster workstation under your desk.
I cannot help but chuckle at how what is old is new again here. In the 70's and early 80's all of IBM's mainframes supported liquid cooling. Basically when a computer "uses" X kW of power it really means that it generates that many kW of heat while it is operating. Removing heat at scale has been a thing for a long time.
And what was alluded to in the video is the thermal mass of 'air' kind of sucks. So the old design of chilling air down to 67 degrees and filling a room with that air so that it can circulate around electronics putting out prodigious amounts of heat, and then collected and re-cooled, is not nearly as efficient as one would like.
Cooling water, piping it to the heat exchanger in the back door of the rack and then (unlike the video's idea) sucking air through it first, and then pushing the cooled air over the electronics to 're-heat' it, works better. Then you don't really care what temperature the air in the data center itself is as long as the heat exchanger can remove 'x' watts of heat from it before it gets blown over the computers. Suck air in from the floor (the coolest air) and blow it out the top (where it continues on to the ceiling).
Still, that only doubles the power capacity of the racks (maybe 2.5x if the heat exchanger is filled with actively chilled water)
Prior to heat exchanger doors people would have "cold" aisles and "hot" aisles. The cold air from the CRAC units would come up from the floor behind the servers, get sucked through them and exhausted forward into the "hot" aisle. There is a whole little mini industry of "cold air containment" which has stuff to build doors/covers for the cold aisle so that all of that air is sucked through servers.
And what was alluded to in the video is the thermal mass of 'air' kind of sucks. So the old design of chilling air down to 67 degrees and filling a room with that air so that it can circulate around electronics putting out prodigious amounts of heat, and then collected and re-cooled, is not nearly as efficient as one would like.
Cooling water, piping it to the heat exchanger in the back door of the rack and then (unlike the video's idea) sucking air through it first, and then pushing the cooled air over the electronics to 're-heat' it, works better. Then you don't really care what temperature the air in the data center itself is as long as the heat exchanger can remove 'x' watts of heat from it before it gets blown over the computers. Suck air in from the floor (the coolest air) and blow it out the top (where it continues on to the ceiling).
Still, that only doubles the power capacity of the racks (maybe 2.5x if the heat exchanger is filled with actively chilled water)
Prior to heat exchanger doors people would have "cold" aisles and "hot" aisles. The cold air from the CRAC units would come up from the floor behind the servers, get sucked through them and exhausted forward into the "hot" aisle. There is a whole little mini industry of "cold air containment" which has stuff to build doors/covers for the cold aisle so that all of that air is sucked through servers.
I'm curious why liquid cooling for computers still uses compression fittings and other odd methods.
Compression fittings are widely out of favor due to their tendency to leak, and nowadays everything is copper, pex, or metal+flared fittings.
I wonder if there's a PC running PEX-A for tubing.
Further thoughts:
Brake lines use flared fittings, with either metal tubing or plastic tubing with metal ends. Uses a special bolt that allows liquid through the pipe to enter the caliper, and is removable.
I would imagine something like that could work to make the lines to the individual servers serviceable without relying on flaky plastic "quick connects".
Further thoughts:
Brake lines use flared fittings, with either metal tubing or plastic tubing with metal ends. Uses a special bolt that allows liquid through the pipe to enter the caliper, and is removable.
I would imagine something like that could work to make the lines to the individual servers serviceable without relying on flaky plastic "quick connects".
RGB-life liquidcooling != Data Center liquid cooling
They have precision dry break quick connects. They use crazy jacketed hoses with PTFE liners and copolymer sheating, sometimes with reinforcment layers. The coolant is pumping through at an operating pressure of 10bar @ 200C. These applications exceed the limits of potable water and hydronic heating systems
They have precision dry break quick connects. They use crazy jacketed hoses with PTFE liners and copolymer sheating, sometimes with reinforcment layers. The coolant is pumping through at an operating pressure of 10bar @ 200C. These applications exceed the limits of potable water and hydronic heating systems
[deleted]
For what reason would a datacenter operate 200C coolant?
The only reasonable thing I can come up with is we got an extra 0 here. 20C/68F sounds like a more practical temperature for keeping the computers alive.
It's actually quite cold, if you're thinking of water blocks on the chips.
It does fit if you're thinking of air-to-water heat exchangers to cool the hot exhaust of a google rack before recirculating.
Something closer to 30C or even 40-45C (the latter if you're in a hot and humid location like Florida and want to get by without refrigeration technology and just increasing the flow rate/heat exchangers) is more realistic for datacenter-scale component-level water cooling.
Yeah, a high-power-density Nvidia chip may need colder water in the 20-30C range, but CPUs and such aren't efficient at the clocks that cause power densities where you need <<40C water.
My 5950X is running at ~30C (return from the radiator) water and 4.5 GHz (maybe even a bit more? I don't recall, the heat wave around here is over) on a torture test with Prime95 where it produces almost double the nominal TDP (thermally limited, due to a few hot-running cores and around 80-85C average core temperature on the others). The base clock is at just 3.4 GHz.
I see phase-change cooling with nucleate boiling as well as very tightly integrated (liquid) water channels in the future, with the latter thinning out the die to weld or solder a better thermal conductor closer to the actual transistor junctions, with minimal distance to the coolant.
Something closer to 30C or even 40-45C (the latter if you're in a hot and humid location like Florida and want to get by without refrigeration technology and just increasing the flow rate/heat exchangers) is more realistic for datacenter-scale component-level water cooling.
Yeah, a high-power-density Nvidia chip may need colder water in the 20-30C range, but CPUs and such aren't efficient at the clocks that cause power densities where you need <<40C water.
My 5950X is running at ~30C (return from the radiator) water and 4.5 GHz (maybe even a bit more? I don't recall, the heat wave around here is over) on a torture test with Prime95 where it produces almost double the nominal TDP (thermally limited, due to a few hot-running cores and around 80-85C average core temperature on the others). The base clock is at just 3.4 GHz.
I see phase-change cooling with nucleate boiling as well as very tightly integrated (liquid) water channels in the future, with the latter thinning out the die to weld or solder a better thermal conductor closer to the actual transistor junctions, with minimal distance to the coolant.
10 bar to the rack that seems a bit high?
How does the coolant get to 200c, that'd imply some sort of inline heatpump right?
How does the coolant get to 200c, that'd imply some sort of inline heatpump right?
Compression fittings are generally more convenient and cost effective than Pex or copper in small volumes and with little experience. The only downside as you note is leaks if someone doesn’t do them right (or as time passes), but that tends to be more of problem at higher household service line pressures (60+ psi)
If it is a low pressure recirculation system, you can get by with almost no tools and most people if they read instructions at all probably won’t mess it up bad enough to have a leak.
The brake line connectors could be interesting, but aren’t likely to work well stock - brakes are very low flow, high pressure as they are ‘dead ended’ hydraulic systems used to transfer force, not loops for cooling purposes. Also those bolts can work loose, and then you have leaks. Brakes require experienced technicians or people die, so folks working on brakes tend to be the more qualified and experienced mechanics, and they make sure to torque to spec, clean surfaces, do proper prep, etc. which is not what you want to have to worry about from an idiot proofing perspective.
If it is a low pressure recirculation system, you can get by with almost no tools and most people if they read instructions at all probably won’t mess it up bad enough to have a leak.
The brake line connectors could be interesting, but aren’t likely to work well stock - brakes are very low flow, high pressure as they are ‘dead ended’ hydraulic systems used to transfer force, not loops for cooling purposes. Also those bolts can work loose, and then you have leaks. Brakes require experienced technicians or people die, so folks working on brakes tend to be the more qualified and experienced mechanics, and they make sure to torque to spec, clean surfaces, do proper prep, etc. which is not what you want to have to worry about from an idiot proofing perspective.
Man, if only copper was easy. One of the gotchas is most connectors for CPU, graphics cards, and chipsets tend to use the G1/4" ends up having metric on the other side. 12mm or 13mm. Most copper pipe is .5", which works out to about 12.7mm. You have to be really picky with couplers if you want to use copper. PTEG/Acrylic is less great for things that would be banged around and a pain, compared to if you know how to actually do copper.
Or you just use actual metric copper pipes in 12mm....
Good luck finding those easily around here!
Yeah, it's one of the more practically relevant downsides of not really switching to metric.
It could start with mandating metric-based norms for building parts in new designs for federally-funded construction, as well as mandating in consumer protection regulation that the metric measurements have to be no less prominent than non-metric units (well, technically that'd be about SI units).
At some point potentially requiring operations with large-enough warehouses to offer the metric-norm equivalents for parts according to non-metric legacy norms, if they reach some threshold with inventory turnover on those parts where it'd not be a big burden to also offer those metric equivalents.
But yeah, I know the feeling of foreign norm components being hard to get, from when I tried to get the screws used for consumer-grade camera tripod mounts. They are iirc 1/4" machine screws. Roughly like M6, but far from compatible.
It could start with mandating metric-based norms for building parts in new designs for federally-funded construction, as well as mandating in consumer protection regulation that the metric measurements have to be no less prominent than non-metric units (well, technically that'd be about SI units).
At some point potentially requiring operations with large-enough warehouses to offer the metric-norm equivalents for parts according to non-metric legacy norms, if they reach some threshold with inventory turnover on those parts where it'd not be a big burden to also offer those metric equivalents.
But yeah, I know the feeling of foreign norm components being hard to get, from when I tried to get the screws used for consumer-grade camera tripod mounts. They are iirc 1/4" machine screws. Roughly like M6, but far from compatible.
Yup, or even equipment with mixed standards that are kinda close so you have to start pulling out multiple thread gauges just to figure out WHY you're particularly screwed, and for how long will you remain so.
Plumbing is rife with it near as I can tell. Semi-random differences in thread pitches, metric vs standard, etc. everywhere.
Plumbing is rife with it near as I can tell. Semi-random differences in thread pitches, metric vs standard, etc. everywhere.
I'm about to get me proper industrial quick disconnects (the hard part is the disconnecting, not the connecting, as otherwise a simple valve on each end and a screwed fitting would suffice) for my workstation, so that I can put a large radiator outside of my room and still have an easy-to-carry system when I need to be somewhere else and have a proper computer. It's just no longer silent at that point.
These quick disconnects are polymer, but that's in large part due to corrosion resistance. They are also rated to ~8 Bar and also vacuum, if one would use the latter.
Also, for flexible tubing, compression fittings don't seem like such a bad idea. Yes, don't use them for rigid pipes (PEX counts there, for connecting purposes), but the only thing better than squishing the hose into a seal-barb seems to be welding the hose to the connector...
These quick disconnects are polymer, but that's in large part due to corrosion resistance. They are also rated to ~8 Bar and also vacuum, if one would use the latter.
Also, for flexible tubing, compression fittings don't seem like such a bad idea. Yes, don't use them for rigid pipes (PEX counts there, for connecting purposes), but the only thing better than squishing the hose into a seal-barb seems to be welding the hose to the connector...
A lot of high power plasma processing equipment is water cooled. RF and DC generators, matches, etc. Stuff that requires a coax cable as thick as your arm. The switch from air to water cooling happens around .5-1Kw, this is the same range they seem to be targeting for datacenter stuff.
Aside from leaks, the building maintenance staff really needs to stay on top of the cooling water chemistry. Cooling loops tend to make slime and if you let it get out of control it clogs up everything and becomes a real problem to clean.
Aside from leaks, the building maintenance staff really needs to stay on top of the cooling water chemistry. Cooling loops tend to make slime and if you let it get out of control it clogs up everything and becomes a real problem to clean.
Liquid cooling also enables heat recovery and free cooling all year long.
Here's a project using liquid cooling to recover energy from Data centers to heat greenhouses.
https://www.qscale.com/
It seems odd to discuss liquid cooling but spend the first part of the article talking about AC PDUs and their issues. If you are able to use liquid cooling in a data center, aren’t you also sophisticated enough for modern DC to point-of-load?
I can see why supermicro needs to pursue this direction, but it seems inevitable that their business will get eaten up by OCP and integration at the rack, row, and building level.
I can see why supermicro needs to pursue this direction, but it seems inevitable that their business will get eaten up by OCP and integration at the rack, row, and building level.
One place i worked used flourinert to cool the circuit cards of a integrated circuit tester.
Each card has a 'cold plate' where the outside of the cold plate has a brass tube brazed to the plate in a serpentine pattern leading to two quick disconnect connectors that automatically engage/disengage when the card in inserted or removed from the card cage.
The other side of the cold plate (the 'inside') has spring loaded fingers to pull the heat away from chips. We had some chips that dissipated over 100W.
This worked great - you'd lose maybe 1 drop of fluid per card insertion, etc. We used flourinert, but DI water, mineral oil or water with ethylene glycol could be used as well.
Each card has a 'cold plate' where the outside of the cold plate has a brass tube brazed to the plate in a serpentine pattern leading to two quick disconnect connectors that automatically engage/disengage when the card in inserted or removed from the card cage.
The other side of the cold plate (the 'inside') has spring loaded fingers to pull the heat away from chips. We had some chips that dissipated over 100W.
This worked great - you'd lose maybe 1 drop of fluid per card insertion, etc. We used flourinert, but DI water, mineral oil or water with ethylene glycol could be used as well.
Technically, the best solution is distribute. Specifically, compute stuff on user's own devices, as opposed to moving everything to these centralized clouds controlled by just a few large corporations.
Too bad that's unlikely to happen.
Too bad that's unlikely to happen.
The "cloud" can be far more energy efficient than personal computing devices, and geographically distributed computation has costs as well, if you're doing computation that requires a lot of data that isn't all in one place.
I think keeping one's own data on one's own local hardware has other significant privacy/security/control benefits, but energy efficiency is probably not among them.
I think keeping one's own data on one's own local hardware has other significant privacy/security/control benefits, but energy efficiency is probably not among them.
More than 50% personal devices are phones and tablets. They are using ARM CPUs notable for power efficiency (because battery powered).
More than 50% of computers being sold are laptops. Guess what, these mobile CPUs are notable for power efficiency, because battery powered.
Only a minority of users (myself included) have desktop computers with FLOPS/watt figure that’s worse than massively multicore server-targeted AMD64 processors used by cloud providers. Clouds are slowly transitioning to ARM64 as well (I have personally shipped a cloud app working on Ampere Altra CPUs) but still, so far majority of them are using faster but less efficient AMD64 CPUs.
Another thing, moving data costs energy. Inside each CPU core, you have like 400GB/sec bandwidth to the caches with no energy overhead. Moving data to/from system RAM takes way more electricity. Moving that data to/from a server on another end of the world takes way more electricity than that: the wires and optical cables are long, and there’re many routers/amplifiers/other hardware in between.
And another thing. Personal devices have limited compute power. This creates incentive for developers to optimize their code. If your stuff is too slow, users with low-end devices will be unhappy. With unlimited compute power of the clouds, it sometimes makes sense to neglect that and just throw resources at the problem.
More than 50% of computers being sold are laptops. Guess what, these mobile CPUs are notable for power efficiency, because battery powered.
Only a minority of users (myself included) have desktop computers with FLOPS/watt figure that’s worse than massively multicore server-targeted AMD64 processors used by cloud providers. Clouds are slowly transitioning to ARM64 as well (I have personally shipped a cloud app working on Ampere Altra CPUs) but still, so far majority of them are using faster but less efficient AMD64 CPUs.
Another thing, moving data costs energy. Inside each CPU core, you have like 400GB/sec bandwidth to the caches with no energy overhead. Moving data to/from system RAM takes way more electricity. Moving that data to/from a server on another end of the world takes way more electricity than that: the wires and optical cables are long, and there’re many routers/amplifiers/other hardware in between.
And another thing. Personal devices have limited compute power. This creates incentive for developers to optimize their code. If your stuff is too slow, users with low-end devices will be unhappy. With unlimited compute power of the clouds, it sometimes makes sense to neglect that and just throw resources at the problem.
What makes this time different?
I've literally heard this is coming to datacenter for 20 YEARS and it hasn't yet.
Liquids like mineral oil, etc to be the cooling agent yet it never gains measured adoption.
And with so much of servers now being centrally managed by just a few major cloud providers (AWS, Azure, etc) - unless you can break into those few accounts - how will this time be any different than the past?
From 2003: https://www.hitachi.com/New/cnews/E/2003/0217/index.html
I've literally heard this is coming to datacenter for 20 YEARS and it hasn't yet.
Liquids like mineral oil, etc to be the cooling agent yet it never gains measured adoption.
And with so much of servers now being centrally managed by just a few major cloud providers (AWS, Azure, etc) - unless you can break into those few accounts - how will this time be any different than the past?
From 2003: https://www.hitachi.com/New/cnews/E/2003/0217/index.html
> Liquids like mineral oil, etc to be the cooling agent yet it never gains measured adoption.
I've never seen a serious attempt to do mineral oil cooling outside of funny gag setups. It was never really pitched as a future of cooling. There's other attempts at other liquids for submersion cooling, but that always seems like a bit of a stretch.
Beyond that, liquid cooling has taken over. That Hitachi post, for example, seems to be about AIOs which are now very common on desktops with the AIO market exploding over the last ~5 years. Nearly every laptop also employs a form of "liquid cooling" these days with either heatpipes or vapor chambers.
As for "what makes this time different" for the datacenter, well heat density continues to rise. >100 CPU cores per 2U rack are trivial to achieve now, and more are readily plausible. GPU accelerators are also more common than ever, and are pushing staggering power draws (like the 8x 400W NVIDIA A100 in a single 4U chasis the article talks about). You can keep spending more power on fans, but there's diminishing returns there not to mention the power consumption of the fans themselves becomes non-trivial.
I've never seen a serious attempt to do mineral oil cooling outside of funny gag setups. It was never really pitched as a future of cooling. There's other attempts at other liquids for submersion cooling, but that always seems like a bit of a stretch.
Beyond that, liquid cooling has taken over. That Hitachi post, for example, seems to be about AIOs which are now very common on desktops with the AIO market exploding over the last ~5 years. Nearly every laptop also employs a form of "liquid cooling" these days with either heatpipes or vapor chambers.
As for "what makes this time different" for the datacenter, well heat density continues to rise. >100 CPU cores per 2U rack are trivial to achieve now, and more are readily plausible. GPU accelerators are also more common than ever, and are pushing staggering power draws (like the 8x 400W NVIDIA A100 in a single 4U chasis the article talks about). You can keep spending more power on fans, but there's diminishing returns there not to mention the power consumption of the fans themselves becomes non-trivial.
Cool stuff. I remember seeing liquid cooling solutions that immersed the entire system in some non-conductive fluid, like mineral oil. Of course, that's a bit messy, but I wonder if all components were designed to be immersed, whether it'd be feasible to consider the entire server enclosure as the watertight unit, e.g. in a rack, rather than hoses to individual servers.
Immersing the rack as a unit means commiting to servicing the rack as a unit.
Some organizations do that. Bring the rack up, do a burn in test, then run it until enough bits are broken that it makes sense to take the whole thing down to swap bits or until it's obsolete enough to justify replacing the whole thing. But a lot of organizations want to replace storage or ram or ?? as it fails without turning off the whole rack.
Would bring more meaning to the words 'draining' a server/rack/data center though.
Actually reading the article, they have immersion chambers which look more or less like a rack rotated 90 to be horizontal. Which means you can pull a rack mount server out vertically to service it, but it also takes up a lot more floorspace (at least 3x to my eye), which reduces density. If you were OK with reduced density, you could leave your racks 2/3rds empty or use 2U servers instead of 1U servers, etc. Power and cooling requirements are already the bottleneck for most datacenters rather than floorspace, and liquid cooling doesn't really help a lot there (unless you can pump your working fluid outdoors, but usually it's just dumping the heat elsewhere in the room more effectively, sometimes with more power for a pump than for fans).
Some organizations do that. Bring the rack up, do a burn in test, then run it until enough bits are broken that it makes sense to take the whole thing down to swap bits or until it's obsolete enough to justify replacing the whole thing. But a lot of organizations want to replace storage or ram or ?? as it fails without turning off the whole rack.
Would bring more meaning to the words 'draining' a server/rack/data center though.
Actually reading the article, they have immersion chambers which look more or less like a rack rotated 90 to be horizontal. Which means you can pull a rack mount server out vertically to service it, but it also takes up a lot more floorspace (at least 3x to my eye), which reduces density. If you were OK with reduced density, you could leave your racks 2/3rds empty or use 2U servers instead of 1U servers, etc. Power and cooling requirements are already the bottleneck for most datacenters rather than floorspace, and liquid cooling doesn't really help a lot there (unless you can pump your working fluid outdoors, but usually it's just dumping the heat elsewhere in the room more effectively, sometimes with more power for a pump than for fans).
I once saw a setup with a whole rack laid down on its back in a tub of mineral oil, like you describe. You could slide individual servers up and set them on the top of the enclosure, letting them drip down into the tub. Kind of messy, but not unreasonably so. (This was setup at SC'09 in Portland. I'm impressed that someone went to the trouble to set this thing up just for a few days of a conference.) The oil was actively pumped through a radiator with a fan.
Part of the whole point of the system is that, since oil is a better coolant than air, they can run the whole system at a higher ambient temperature, like 80 or 90 degrees F instead of 70-ish. That could reduce energy costs quite a bit.
I'm not sure if the oil goes bad eventually and has to be replaced; I could see that being expensive if there isn't a simple way to clean/recycle/reprocess it back into usable oil.
Part of the whole point of the system is that, since oil is a better coolant than air, they can run the whole system at a higher ambient temperature, like 80 or 90 degrees F instead of 70-ish. That could reduce energy costs quite a bit.
I'm not sure if the oil goes bad eventually and has to be replaced; I could see that being expensive if there isn't a simple way to clean/recycle/reprocess it back into usable oil.
In other words, HPC
Submersion cooling systems are shown on page 3 of the article. The page selector is at the bottom of the text. I like how they have to have hoists to raise and lower the servers into the bath because they're so heavy.
Cool, I should have read more of the article.
Direct liquid cooling makes a shitload of sense from a purely academic standpoint.
At what point in power density would it not even matter if the datacenter was at meat locker temperatures? I feel like we are getting pretty damn close.
The efficiency gains at scale must be really good. I can see why this is not super popular though.
At what point in power density would it not even matter if the datacenter was at meat locker temperatures? I feel like we are getting pretty damn close.
The efficiency gains at scale must be really good. I can see why this is not super popular though.
Data centers have been using chilled water to cool air for a long time. It would be interesting to see cooling delivered directly to the CPU/GPU from a/the chiller.
No thanks, the equinix DC I have dealt with has had so many chiller outages I am glad that the fans were able to keep the temp low enough to prevent damage.
If I lost all cooling the equipment would literally be toast.
If I lost all cooling the equipment would literally be toast.
Almost no server equipment will be damaged by loss of cooling; it will shut down before damage occurs.
Repeat overheating even with auto shutoff can result in solder joints going bad and reducing the life span of hw. Additionally not all components that are overheating will cause on auto shutoff.
I think datacenters will opt for more ARM architectures before introducing liquid at scale. Things like Gravitron and Apple Silicon are changing the landscape there.
Graviton and Apple Silicon still don't offer significant performance per watt improvements versus EPYC. I don't think it will make or break the need for liquid cooling.
The article is mostly about cooling high power, GPU-like accelerator units that consume 400-600W or more.
Even with power efficient architectures, the push to scale up core count and increase density will still drive toward more liquid cooling.
Even with power efficient architectures, the push to scale up core count and increase density will still drive toward more liquid cooling.
You still have a 280W CPU, instead of 64 Core you now have 128. And if thermals permits they would like 128 x86 Core or 256 ARM Core.
Samsung just announced their work of 7xxGB of DDR5 per stick. With a future roadmap of 1TB per stick at a request of hyperscaler. Along with 400W or 500W GPU.
And when you have many new packaging technology where each die could be a 100W TDP you quickly run into all sort of thermals limits. And there are all sort of HPC and niche that wants super powerful Server.
Samsung just announced their work of 7xxGB of DDR5 per stick. With a future roadmap of 1TB per stick at a request of hyperscaler. Along with 400W or 500W GPU.
And when you have many new packaging technology where each die could be a 100W TDP you quickly run into all sort of thermals limits. And there are all sort of HPC and niche that wants super powerful Server.
I’m pretty sure AMD offers perf/watt that is in the ballpark of ARM chips, and I’m guessing Intel, if they ever get on a similar process node again, will be able to get close.
None of that changes the need for accelerators though. Nvidia et al want to make the biggest chips they can, and at some point that means water cooling.
None of that changes the need for accelerators though. Nvidia et al want to make the biggest chips they can, and at some point that means water cooling.
Is there a reason the ai accelerators etc can’t just be parked in the polar circle?
Not every workload is that latency sensitive presumably
Not every workload is that latency sensitive presumably
> need to handle cooling at >1kW/U
I wonder if this waste heat could be used to power active/phase change cooling.
I wonder if this waste heat could be used to power active/phase change cooling.
The challenge is that it's really low grade heat (< 100C). DC waste heat would probably find its best use in district heating applications (though I did read about some cases still using a heat pump to boost the waste heat stream even higher - still for district heating though).
Why is that guy wearing a mask? Either servers are infectious or wearing a mask is now just "expected".
Do any of you do a doubletake when you see people on tv without a mask?
Have you noticed characters in cartoons wearing masks? Check out the weather frog, sitting alone in the middle of a forest, wearing a mask.
This how we become Morlocks.
Do any of you do a doubletake when you see people on tv without a mask?
Have you noticed characters in cartoons wearing masks? Check out the weather frog, sitting alone in the middle of a forest, wearing a mask.
This how we become Morlocks.
I mean yeah, but I doubt its going to be mainstream anytime soon.
At the moment its cheaper to just use forced air, and yolo it. Running at half density is expensive, but not as expensive as backfilling everything with liquid cooling.
Also given that we've been able to run 2 socket blades at full bore without liquid cooling kinda suggests its not actually needed.
Having radiators on the front/back of the rack directly works really well. We had it on our render farm. combine that with enforced hot isle/cold isle and you can reduce the need for aircon dramatically, without multiplying your leak risk by at least 96 times.
The massive problem here is that its really difficult to hotswap anything. Those cooling pipes need to be removed before you can pull out the server. Unless they are self sealing (like hydrolic lines) then you need to drain the loop first. That costs a shitload of money at scale.