By Raffaele Luca Amalfi, Ph.D., Jackson Braz Marcinichen, and John Richard Thome, Ph.D.
CONTEXT AND NEW COOLING TECHNOLOGY
Heat dissipations of servers and their racks in datacenters are reaching ever increasing levels, breaching the economical heat removal limits of traditional air-based cooling technologies. Currently, about 40% to 45% of the total datacenter energy consumption is used to cool servers with an annual growth rate of about 15%, presenting significant challenges to keep reasonable values for energy efficiency and noise level within bound [1, 2]. Thus, there is an immediate need to improve the rational use of energy in these systems by implementing a novel passive two-phase thermosyphon cooling technology, as reported in Figure 1.
Figure 1 . Concept of the new proposed cooling technology for high heat dissipation servers [3].
The envisioned cooling technology operates with numerous mini-thermosyphons (one per server) in order to dissipate the heat produced by memories, converters, microprocessor, etc. into the evaporator cold plates and out through the two-phase micro-condensers [4]. Then, the heat is transferred to a macro-thermosyphon with a compact plate heat exchanger (PHE) located on top of the rack (see Figure 1) acting as the condenser, which dissipates the total heat from the rack (60 kW or much more) into the datacenter’s water cooling loop. This approach provides a passive and low energy consumption cooling solution coupled with the possibility to re-use the waste heat from the datacenter (i.e. district heating network), but that is beyond the scope of the present study.
THERMOSYPHON OPERATING WITH A PHE
In general, a thermosyphon is a closed and self-regulated system composed of four main components: an evaporator, a riser, a condenser and a downcomer (see Figure 2). The condenser is always installed above the evaporator, while the riser and downcomer connect them together, allowing the working fluid to circulate within the loop.
Figure 2. Schematic of a thermosyphon loop and a brazed PHE is proposed as a water-cooled condenser to transfer the heat from the entire datacenter to the recovery utility.
More specifically, the heat generated by the electronics is removed in the evaporator, where two-phase flow is generated. Next, the liquid/vapor mixture, with a lower density, is guided up to the condenser by the riser due to buoyancy effects, where it condenses back to 100% liquid. The subcooled liquid, with a higher density, is then guided to the evaporator by the downcomer thanks to its gravity force, in order to start the loop again. The fluid flow within a thermosyphon loop is self-sustained and does not require any mechanical drivers or controllers. In operation, the thermosyphon seeks an equilibrium state defined by the momentum balance between the static, momentum and frictional pressure drops along the entire loop.
In the cooling system depicted in Figures 1 and 2, a brazed PHE is proposed as an overhead condenser, operating in counter-flow arrangement, with the working fluid condensing on the prima- ry side (at the inlet vapor quality from the riser) and datacenter water on the secondary side. PHEs represent an important cate- gory of compact heat exchangers widely used for many applications. These units consist of thin, rectangular, pressed stainless steel plates that are stacked together, such that cold and cold fluid streams alternate through the inter-plate passages. The plates are stamped with various corrugation patterns (washboard, oblique washboard, chevron, etc.), providing about 10-20% increase in effective heat transfer area compared to the footprint, and additionally, modify the flow field to promote more turbulence and mixing with higher heat transfer performance [5]. In particular, Figure 3 shows a schematic of a plate with a chevron corrugated pattern and its associated geometrical parameters defined.
Figure 3. Geometrical parameters of a sinusoidal chevron corrugated pattern with inlet and outlet ports located at the same side [6].
The present study will show a case study simulation of a compact PHE to recover 61 kW from a datacenter rack. These units are easily adaptable to datacenter thermal-hydraulic requirements by changing the number of plates and/or modifying the corrugated pattern and/or plate size. Their compact design has a low weight and a reduced working fluid charge.
They are commercially available from numerous suppliers. However, as their typical application is refrigeration systems, the thermal design simulation codes used for those applications with superheating, desuperheating and lubricating oil effects are not usually accurate for the present partial condenser case in datacenters. Due to its accuracy to locally model two-phase heat transfer mechanisms, our PHE simulation code, which is described below, is used here for predicting thermal-hydraulic performance of a datacenter overhead water-cooled condenser, as reported in Figure 1.
PHE SIMULATOR
An in-house simulation code was developed to rate and design refrigerant-to-water chevron PHEs in single- and multi-passage configurations. The present simulator was entirely written in Python and it is able to model evaporation, condensation and single- phase (liquid or vapor) flows on the primary side, while the secondary side is water in single-phase liquid flow.
Furthermore, the simulator includes desuperheating of vapor (both dry-wall and wet-wall local conditions) for flows exiting a compressor in a refrigeration cycle. In addition, it can handle several flow configurations (parallel-flow and counter-flow), diffe- rent flow orientations (horizontal and vertical) and many plate arrangements, geometries and working fluids. Also, it can handle partial condensers, which are needed in the present study. The code can be classified as a “pragmatic” tool, being able to provide accurate results and guarantee very low computational time (in order of seconds to complete a fully detailed local incremental thermal-hydraulic simulation of a PHE, including its pressure drops with their inlet and outlet port pressure drops).
The local one dimensional Effectiveness-NTU method [7] , which includes the equations of mass, momentum and energy conservation, was implemented and coupled with a set of appropriate correlations to predict the local heat transfer coefficients and lo- cal frictional pressure gradients for single- and two-phase flows along the flow in the PHE (more details on the modeling and simulator’s flowchart can be found in [8]). It has to be pointed out that the best proven methods for local thermal-hydraulic performance of evaporating flow within PHEs were developed in [9] using an experimental databank of 3416 points, including our data [6, 10, 11] and additional data from 12 other research labs worldwide. The most reliable prediction methods for condensing flow were determined [12] through a state-of-the-art research study that compared the existing prediction methods against another large experimental condensation heat transfer and pressure drop databank from literature. The code has an incremental simulation approach in which the PHE is divided into numerous small control volumes from inlet to outlet, which are then added together to get the total system performance.
Finally, the current simulation tool can be easily managed by the users, thanks to a user-friendly and dynamic GUI, which includes a complete description of the input and output parameters, provides comments from the code and error handling in case of user’s mistakes or introducing input values out of range.
CONDENSER CASE STUDY AND SIMULATION RESULTS
This section presents a case study simulation for a compact PHE used as a water-cooled condenser of the macro-thermosyphon depicted in Figure 1, chosen here to dissipate 61 kW from a datacenter rack to the utility water. The macro-thermosyphon used refrigerant R134a as the working fluid. The simulation conditions are summarized in Table 1, while the geometry of the PHE is reported in Table 2.
Figure 4 presents the local temperature profiles for the refrigerant (red curve) and the water (blue curve) as a function of the PHE incremental length. The refrigerant entered the PHE with an inlet vapor quality of 40%, and then was completely condensed and subcooled along the length of the PHE. The refrigerant saturation temperature decreased along the PHE due to its two-phase pressure drop, accounted for within the NTU temperatures. The single-phase water received the heat and its temperature increased from 20°C to 48°C at the outlet of the PHE (this means that the datacenter cooling water at 20°C is sufficient here, since higher temperatures can be used by increasing the size of the PHE).
Figure 4. Local temperature profiles for primary (R134a, red curve) and secondary (water, blue curve) sides as a function of the PHE incremental length (1-D discretization using 49 control volumes).
Figure 5 illustrates the local trends of the heat transfer coefficients for both sides. In particular, the water side (blue curve) heat transfer coefficient increased from the inlet to the outlet due to its temperature rise, which provided an increase in thermal conductivity; the refrigerant side (red curve) heat transfer coefficients were higher in the two-phase flow region as expected, compared to the fully single-phase liquid region. It has to be pointed out that the discontinuity at low vapor quality (0%) is due to the transition from two-phase flow to single-phase liquid flow, as well as the use of different prediction methods to estimate the thermal-hydraulic performance in those regions.
Figure 5. Local heat transfer coefficients for primary (R134a, red curve) and secondary (water, blue curve) sides as a function of the PHE incremental length (1-D discretization using 49 control volumes).
Based on the simulation inputs reported in Tables 1 and 2, the pre-sent PHE is able to dissipate 60.6 kW with an overall heat transfer coefficient of 1210.6 W/m2/K. The associated total pressure drops for the refrigerant and water were calculated to be 54.8 kPa and 8.7 kPa respectively. In an actual application, the PHE design would be optimized to reduce its refrigerant side pressure drop, so as to increase the thermosyphon’s flow rate.
CONCLUSIONS
In the present article, a passive thermosyphon cooling scheme has been proposed for cooling of high power datacenter racks using a PHE as the overhead water-cooled condenser. The modeling of the PHE was performed by using our in-house simulator that incorporates the local effectiveness-NTU method with the mass, momentum and energy equations, coupled with the best proven methods from literature to calculate the local heat transfer coefficients and the local pressure drops. The simulation results demonstrated the potential to remove 61kW of heat from a rack by installing a small PHE (only 286 mm x 60 mm x 145 mm), with the possibility to go even higher in terms of power by properly modifying the PHE design.
REFERENCES
[1] P. Jones, Energy Prices Likely to Predict Key Data Center Market Moves, Analyst Says (www.Datacenterdynamics.Com/News/Energy-Prices-Likely-to-Predict-Key-Data- Center-Market-Moves-Analyst-Says/74618.Article). Datacenter Dynamics The Business of Data Centers, 2013.
[2] J. Utsler, Computing in 3-D (www.Ibmsystemsmag.Com/Mainframe/Trends/Ibm-Research/Computing-in-3-D/). IBM Systems Magazine, in press., 2010.
[3] S. Szczukiewicz, N. Lamaison, J.B. Marcinichen, J.R. Thome, P.J. Beucher, Reviewed: Passive Thermosyphon Cooling System for High Heat Flux Servers. ASME 2015 International Techni- cal Conference and Exhibition on Packaging and Integration of Electronic and Photonic Microsystems collocated with the ASME 13th International Conference on Nanochannels, Microchannels, and Minichannels, 2015, pp. V003T010A009.
[4] C.L. Ong, R.L. Amalfi, N. Lamaison, J.B. Marcinichen, J.R. Thome, Two-Phase Mini-Thermosyphon for Cooling of Datacenters: Experiments, Modeling and Simulations. InterPACK 2017 Proceedings of the ASME 2017 International Technical Conference and Exhibition on Packaging and Integration of Electronic and Photonic Microsystems, 2017, pp. 74030.
[5] R.L. Amalfi, F. Vakili-Farahani, J.R. Thome, Flow Boiling and Frictional Pressure Gradients in Plate Heat Exchangers. Part 1: Review and Experimental Database. Int. J. Refrigeration, 2015, pp. 1-19.
[6] R.L. Amalfi, J.R. Thome, V. Solotych, J. Kim, High Resolution Local Heat Transfer and Pressure Drop Infrared Measurements of Two-Phase Flow of R245fa within a Compact Plate Heat Exchanger. International Journal of Heat and Mass Transfer Vol. 103, 2016, pp. 791-806.
[7] T.L. Bergman, A.S. Lavine, F.P. Incropera, D.P. Dewitt, Fun- damentals of Heat and Mass Transfer (Seventh Edition). John Wiley & Sons, Inc., 2011, pp. 1-1048.
[8] R.L. Amalfi, J.R. Thome, Two-Phase Flow Simulations within Plate Heat Exchangers. ITHERM: Seventeenth Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems, San Diego (USA), 2018, pp. 105.
[9] R.L. Amalfi, F. Vakili-Farahani, J.R. Thome, Flow Boiling and Frictional Pressure Gradients in Plate Heat Exchangers. Part 2: Comparison of Literature Methods and New Prediction Methods. Int. J. Refrigeration, 2015, pp. 1-19.
[10] F. Vakili-Farahani, R.L. Amalfi, J.R. Thome, Two-Phase Flow and Boiling of R245fa in a 1mm Pressing Depth Plate Heat Exchanger – Part 1: Adiabatic Pressure Drop. Interfacial Phenomena and Heat Transfer Vol. 2, 2014, pp. 325-342.
[11] F. Vakili-Farahani, R.L. Amalfi, J.R. Thome, Two-Phase Flow and Boiling of R245fa in a 1mm Pressing Depth Plate Heat Exchanger – Part 2: Flow Boiling Heat Transfer. Interfacial Phenomena and Heat Transfer Vol. 2, 2014, pp. 343-361.
[12] F. Vakili-Farahani, R.L. Amalfi, J.R. Thome, Two-Phase Heat Transfer and Pressure Drop within Plate Heat Exchangers. Encyclopedia of Two-Phase Heat Transfer and Flow II Vol. 2, 2015.
ABOUT THE AUTHORS
Raffaele Luca Amalfi
Dr. Raffaele Luca Amalfi is a Scientific Collaborator at the Swiss Federal Institute of Technology of Lausanne (EPFL), at the Laboratory of Heat and Mass Transfer (LTCM), in Switzerland. He holds a Ph.D. in Mechanical Engineering from the EPFL. In 2016, he joined Nokia Bell Laboratories in New Jersey, where he performed disruptive research on hybrid air/liquid cooling technologies for shelf-level equipment. In 2012, he joined IBM Research Laboratory in Switzerland, where he worked on a novel cooling system for high-power electronics. His research activities and interests include: energy recovery systems, single- and two-phase flow heat transfer within complex structures, heat exchangers design and rating, electronics cooling and passive cooling technologies. He has authored over 20 articles, including book chapters, journal and conference papers.
Jackson Braz Marcinichen
Dr. Jackson Braz Marcinichen is the founder and CEO of JJ Cooling Innovation SÀRL (Lausanne-Switzerland) and has over 25 years experience in HVAC & R systems. He has authored over 60 scientific and technical papers in indexed journals and international peer-reviewed conferences, book chapters and US patents. All along his career, he has computationally simulated, designed and evaluated several experimental facilities and prototypes, characterizing the thermo-hydrodynamic and control of cooling systems (calorimeters, wind tunnels, hybrid systems, etc.).
John Richard Thome
Prof. John Richard Thome is Professor-Emeritus of Heat and Mass Transfer at the École Polytechnique Fédérale de Lausanne (EPFL), Switzerland since 1998 (obtaining his PhD at Oxford University in 1978). Having retired in July 2018, he has now founded a new consulting/thermal engineering software company, JJ Cooling Innovation Sàrl in Lausanne. He is the author of five books and over 245 journal papers on microscale and macroscale two- phase flow, boiling/condensation heat transfer and micro-two-phase cooling systems for electronics cooling, including numerous sponsored projects with IBM, ABB, Nokia Bell Labs, etc.
He received the prestigious 2017 Nusselt-Reynolds Prize, the ASME Heat Transfer Division’s Journal of Heat Transfer Best Paper Award in 1998, the United Kingdom’s Institute of Refrigeration J.E. Hall Gold Medal in 2008 for his work on microscale refrigeration heat transfer, the 2010 ASME Heat Transfer Memorial Award, the ICEPT-HDP 2012 Best Paper Award on a 3D-IC prototype with in- terlayer cooling (13,000 TVS’s and 260 microchannels inside), the ASME Journal of Electronics Packaging Best Paper Award in 2014, and the Outstanding Paper Award at InterPACK2017.
He founded the Virtual International Research Institute of Two-Phase Flow and Heat Transfer in 2014, now with 25 participating universities to promote research collaboration and education (see http://2phaseflow.org). According to Google Scholar, he has over 21,000 citations.
