This blog summarizes some of the recent findings from an ongoing project “An In-Depth Understanding of Oil Immersion Cooling Strategies for Data Centers” at the University of Texas at Arlington (UTA). This project is supported by NSF I/UCRC grant in the program “Energy-Smart Electronic Systems” (ES2) and by many industry partners.
Complete immersion of servers in electrically nonconductive oil is well-known to be a promising technique for minimizing cooling energy consumption in data centers. Liquid cooling in general offers significant advantages over traditional air cooling approaches due to the higher heat capacities of fluids. Although oils have lower heat capacity than water, oil cooling has advantages over other types of liquid cooling methods in that it is a dielectric liquid and does not require complex infrastructure within individual servers. However, lack of sufficient published data and long term reliability data on oil cooling makes most data center operators hesitant to apply these approaches to their mission critical facilities. Initial testing and data collected at UTA on a single server immersed in oil show that it is possible to use high temperature (~45°C) and low flow rate oil to cool a server, requiring minimal cooling energy.
The purpose of this project at UTA is to explore and develop a deeper understanding of the concept of submersion cooling of information technology equipment (ITE). The goals of this project can be broken down into two primary areas:
- The cooling efficiency of oil immersed ITE
- The operational efficiency (overall life cycle of oil, reliability of components, and serviceability) of an oil cooled data center)
Figure 1 below shows three 2U (rack unit, 1.75”x2) servers used as a test vehicle for oil immersion cooling studies at UTA.
In the first of our findings published in 2014 as “Flow Rate and Inlet Temperature Considerations for Direct Immersion of a Single Server in Mineral Oil”(Richard Eiland, John Fernandes, Marianna Vallejo, Dereje Agonafer, Veerendra Mulay), we examined the thermal characteristics of a single server fully submerged horizontally in mineral oil. Experiments were conducted to observe the effects of varying the volumetric flow rate and oil inlet temperature on thermal performance and power consumption of the server. Specifically, temperature measurements of the CPUs, motherboard components, and bulk fluid were recorded at steady state conditions. Comparing with results from baseline tests performed with traditional air cooling show promise for mineral oil as a viable cooling alternative for data centers. Overall, the cooling loop was able to achieve partial power usage effectiveness (pPUECooling) values as low as 1.03 (Electronics Cooling published an article on various power usage efficiency metrics in the data center last year). The server level study provides a preview of possible facility energy savings by utilizing high temperature and low flow rate oil for cooling. Readers of this blog are encouraged to review the discussion on additional opportunities for optimization of IT hardware and implementation of oil cooling in the article.
In the second of our findings published in 2016 as “Effects of Mineral Oil Immersion Cooling on IT Equipment Reliability and Reliability Enhancements to Data Center Operations”(Jimil M. Shah, Richard Eiland, Ashwin Siddarth, Dereje Agonafer), we presented a review on the changes in physical and chemical properties of information technology (IT) equipment materials like polyvinyl chloride (PVC), printed circuit board (PCB) and switching devices when submerged in mineral oil with emphasis on the reliability of interconnects.
Improved efficiency and simplicity in the data center facility’s design could result when mineral oil-based cooling is employed and may even provide some cost savings. But this has not encouraged vendors of existing air cooling solutions to switch. Compelling physics of direct immersion cooling alone is not enough for data center operators because there are uncertainties and concerns persisting on the effects of mineral oil-immersion on reliability. Among these, changes in properties of mineral oil like kinematic viscosity and dielectric strength for example are cited as important factors and are discussed more in depth. Changes in mechanical properties like elasticity, hardness, swelling, and creep behavior have also been documented. The chemical reaction between solid materials and mineral oil as a function of time and temperature was also investigated.
In fact, oil immersion cooling of data center appliances offers opportunities for reliability enhancement as it minimizes common operational issues like overheating and temperature swings in the system; there will be no occurrences of fan failures in servers, noise, dust, air quality concerns, corrosion, electrochemical migration, and occurrence of whiskers. The foregoing are some of the common problem areas when air cooling is deployed in data centers.
In our third findings published in 2016 as “Design Considerations Relating to Non-Thermal Aspects of Oil Immersion Cooling”(Jimil M. Shah, Syed Haider I. Rizvi, Indu Sravani Kota, Sahithi Reddy Nagilla, Dhaval Thakkar, Dereje Agonafer), we explored more material property considerations such as the chemistry, flammability, material compatibility, human health effects, and the sustainability of mineral oil as a cooling liquid. Flammability of oils is a valid concern when immersing costly IT equipment generating heat. The evaporation loss of the cooling oil is also a concern as it impacts maintenance costs. Leeching of materials especially from plastics is a recurring concern in the mineral oil immersed IT equipment.
The long-term stability and performance of key material parameters of oils used in applications such as high voltage power were monitored. This paper also includes results of the Cup Burner Experiment as per ISO 14520/NFPA 2001 standard to determine the minimum design concentration of the fire extinguishing agent for the class B hazard of heavy mineral oil and the class C hazard of electronic equipment. The visual observations of the servers after immersion in oil for 8 months were also documented.
In closing, oil immersion cooling is well on its way to replace air-cooled thermal management solutions in data centers. However, some of the problems still remain –disk drives for example must be completely sealed or use solid state drives. Here at UTA, we are currently working on an accelerated life testing of submerged sealed helium-filled HDDs including the effects of positive and low pressures on the performance and health of HDDs at different altitudes. The oil impact on signaling and communication to and from a server through an evaluation on optics in an oil immersed server is also a latest ongoing research at UTA. There are larger concerns such as warranties on IT equipment provided by vendors which are typically neither covered nor made for oil immersion cooling. There are also cost concerns on the mineral oil itself but when put in larger context of data center cooling costs, it could be justified. Finally, there needs to be business model innovations covering capex and opex costs that make it attractive for data centers to switch to oil immersion cooling.
