As anyone who has ever jumped into a body of water during a summer heat wave will tell you, submerging yourself in cool (or even warm) water is a great way to cool down. This common-sense insight has not been lost on mechanical and thermal engineers charged with cooling servers, storage and network communications devices. In fact, all the way back in 1899, the first patent concerning the use of oil as a coolant and insulator for transformers was granted.
Given that the electricity used by computers has a 1:1 ratio with heat, it’s not surprising that over 50 years later, as the computer industry was in its infancy, IBM patented a method for immersing computer components in dielectric fluid to cool them. Since those days, approaches to immersion cooling have continued to evolve. In this post, we’ll provide a brief overview of immersion cooling, the different methods in practice and the relative strengths and weaknesses of these methods.
What Is Immersion Cooling?
To date, the most common method for cooling IT hardware (from laptops to the data center) has been air. But the fans, ductwork and HVAC systems required to cool data centers with air consume a lot of space and electricity. Using fluid to cool IT hardware, such as one might find in cutting-edge gaming rigs or high performance computing clusters, requires pipes, pumps and a significant amount of space. Plus fans are still needed to deal with the heat that the ‘cold plates’ and ‘heat pipes’ can’t address.
Because immersion cooling is more energy- and space-efficient, interest in the technology has steadily grown over the years, and the technology itself has evolved considerably. But what is immersion cooling?
As the phrase clearly implies, immersion cooling involves immersing system components, such as a motherboard or an entire computer system, in a fluid which, ideally, has a high coefficient of heat rejection and low thermal resistance. The process requires the use of fluids that will not damage the IT components or degrade system function. Further, for safety reasons, these fluids need to be “dielectric,” meaning they do not conduct electricity. The dielectric fluids used for immersion cooling today fall into two categories: oils (synthetic, mineral, bio) and engineered fluids, such as 3M’s Novec or Fluorinert lines.
Single-Phase Immersion Cooling vs. Two-Phase Immersion Cooling
There are two basic approaches to immersion cooling.
The first approach is referred to as single-phase immersion cooling. In this approach, the components are immersed in a dielectric fluid, typically an oil or engineered fluid of some kind. The heat generated by the IT components is absorbed by the fluid and then the fluid is pumped and circulated around an enclosure, chassis or tank to help remove the heat. This entails pumping out the hot oil/fluid to be cooled by a secondary air-to-liquid or liquid-to-liquid heat exchanger, and pumping cooled oil/fluid back into the immersion bath.
A second approach takes advantage of the heat rejected through phase change. As with single-phase, the IT components are immersed in a dielectric fluid, but the fluid used in this case is engineered to have a boiling point which is below the temperature of heat-emitting IT components like CPUs, GPUs, ASICs, power supplies, DC/DC converters and more. In essence, when the IT gear is operating, the heat is removed by a liquid-to-gas phase change.
The interesting thing about phase change is that, once the boiling point is reached, the dielectric fluid itself doesn’t get any hotter. Rather than continually raising the temperature of the dielectric fluid, the heat is rejected when the vapor gas comes into contact with a specially designed vapor-to-liquid heat exchanger that is inside of a specially designed DataTank™. Since the heat exchange happens inside the DataTank, there is no need for a secondary heat exchanger and pumping system. This eliminates a potential point of failure and drastically lowers the complexity and cost of the immersion cooling system.
The interesting thing about 2-phase immersion cooling is that it consumes little to no energy in and of itself. The rising gas caused by the phase change inside the DataTank condenses on the tubes (heat exchangers) inside the top of the DataTank, then changes back to a liquid in the form of small droplets, which fall back into the liquid ‘bath’, as the whole process begins anew.
Immersion Cooling Implementation Methods
There are also two basic methods for implementing immersion cooling.
The first method employs an entirely enclosed IT chassis. The customized chassis provides containment of the dielectric fluid and oftentimes less fluid is needed as a result. Part of the appeal of this method is that it allows self-contained, immersion-cooled chassis to be installed in conventional server racks. A Coolant Distribution Unit (CDU) can likewise be used to manage the coolant across multiple chassis. A major challenge of this approach is that the immersive chassis must be replaced with every IT refresh, an average of 10 complete replacements over the usable life of the IT rack.
The second method employs an entirely enclosed IT tank, or DataTank. The tanks provide containment of the dielectric fluid and are typically designed to accommodate IT gear that would otherwise be mounted in 19”, 21” or OCP style racks for example. Since the DataTanks accommodate almost all types of IT gear, an IT refresh is simply out with the old and in with the new — meaning there is no need to replace the DataTank for an IT refresh. This makes the flexibility, cost and TCO highly appealing for DataTanks versus the chassis approach, whereas the chassis approach enjoys the benefit of some dielectric fluid volume savings.
Immersion Cooling and the Sustainable Data Center
As you can see from this brief description, there are several major differences between single-phase immersion and two-phase immersion.
First, single-phase relies on pumping an oil or dielectric fluid to a secondary heat exchanger and pumping system (Coolant Distribution Unit or “CDU”) which then rejects the heat to the building’s primary water heat rejection loop. There are some one-phase immersion systems with heat exchangers located inside of a specially engineered IT chassis, which reduces the need for a CDU. In either case, the oil or dielectric fluid needs to be pumped across substantially large IT heat sinks on the server boards because the process does not benefit from the higher heat rejection capacity of phase change. Using mineral oil or synthetic petrochemicals can involve a lot of messy cleanup if you ever need to swap out IT gear, and is therefore a non-starter for some potential users.
Further, most of these fluids have a flash point, meaning they are flammable, and this represents a potential hazard and risk in data center operations.
Finally, because single-phase immersion relies on additional pumps to circulate and cool the oil, the energy efficiency gains made by shifting from air to immersion cooling are reduced. That is not to say that 1-phase immersion cooling isn’t beneficial; it certainly provides dramatic energy efficiency and IT heat load densities versus air cooling.
The benefits of one-phase immersion cooling include:
- Better energy efficiency than air cooling
- About 10X heat rejection capacity vs. air cooling
- Mineral oil is less expensive than 2-phase engineered dielectric fluids
- Oils generally do not evaporate (however, ‘oil blooms’ are generally experienced within a 1-2 meter radius of most single-phase immersive tanks/enclosures)
- Lower CAPEX than air cooling in some cases
- Less space required versus air cooling
- Better TCO than air cooling in some cases
- Lowers or eliminates the use of water for outside heat rejection
- Quiet operation
The benefits of two-phase immersion cooling include:
- Best known efficiency in any form of cooling
- 2X (or greater) heat rejection capacity vs. one-phase
- Half the space requirement versus one-phase (no bulky heat sinks or CDUs)
- Lower CAPEX than air cooling (per kW)
- Lower TCO than air cooling (per kW)
- Waste heat can be re-used for hot water, district heating or energy generation
- Dielectric fluids are clean and make servicing or replacement of IT gear simple
- Faster builds than air-cooled data centers
- Lowers or eliminates the use of water for outside heat rejection
- Silent operation
Because immersion cooling, particularly two-phase immersion cooling, is so energy efficient, many see it playing a central role in the evolution of the sustainable data center. If you would like to learn more about that, we invite you to read our white paper, Liquid Cooling: The Key to Data Center Sustainability.
If you would like more information on LiquidStack’s two-phase immersion cooling systems, please get in touch.