Lance Dumigan
Fabrisonic, LLC
Mark Norfolk
Fabrisonic, LLC
Practical Knowledge Gained
“The performance of DTC cold plates can be enhanced by a new method of fabrication called Ultrasonic Additive Manufacturing (UAM). Building DTC cold plates with UAM could enhance performance and functionality.”
The direction of Artificial intelligence and High Power Computing (AI/HPC) is driving greater computation and power requirements, resulting in greater heat generation. With the emergence of Large Language Models (LLM), computational power is more important than ever. A.I. that generates text, images, and media requires larger computational power, memory, and higher infrastructure costs. [1] The current trend for AI/HPC servers, with a power usage of 20-40kW [2] per rack, challenges conventional air cooling technologies. AI/HPC data centers have been moving toward Liquid Immersion (LI) and Direct-To-Chip (DTC). LI is the emersion of a rack, or cluster, into a liquid, and DTC is attaching a water-cooled cold plate that extracts heat directly from the electronic device (GPU/ CPU). Of those two options, DTC would lend itself to a new build and a retrofit to existing servers.
As these servers see higher power and density increases, they must be met with an increase in cooling capacity. Data center operators must install the infrastructure to accommodate this new dynamic. These higher densities are projected to be as high as 50–100 kW per rack. DTC cold plates must meet these higher heat extraction requirements. There is greater pressure on DTC cooling to extract heat from CPUs, GPUs, and memory. According to Gartner Research, Rack power densities could reach as high as 100kW as A.I. models and synthetic data need greater processing capacity. [3] Looking further into the future, primary and secondary loop water pressure, flow, and even capacity will be considerations that will need to be addressed. While increasing the diameter of the manifold 2x would enhance the flow distribution by as much as 10% [4], those looking into the future of data centers see water conservation as a growing priority [5]. With the cost of modern processors approaching $15k per device (or higher), optimizing their performance is essential when multiplying the number of processors per server and the number of servers per rack.
Considerable research and discovery have been performed concerning the DTC option. The cooling plate flow distribution, flow rate, thermal hot spots, temperature uniformity, microchannel plates, and different flow schemes have been evaluated [6]. Companies such as JetCoolTM, AsetekTM, SupermicroTM, and many others have products for a DTC solution. However, a highly relevant and little-known additive manufacturing (AM) process is poised to enhance the effectiveness of the existing DTC technologies. The AM process that offers this enhancement is referred to as Ultrasonic Additive Manufacturing (UAM) [7]. UAM offers several capabilities that could be employed to accelerate the extraction of (energy) heat in a DTC cold plate. As UAM can combine dissimilar metals, embed sensors and materials, and print cold plates from a few inches to a few meters with complex internal geometries with smooth (machined) passages, this technology’s application represents the potential for significantly increased DTC cold plate performance.
Most people are familiar with Additive Manufacturing (AM). The most widely known forms of AM use a metallic powder that is solidified through the addition of heat energy (laser, for example), which melts the powder in-plane and builds the shape by melting one layer of powder upon the other. Other methods include using an adhesive or binder to create a shape for each layer and then building successive layers to create the shape (product). UAM uses metal foils in a sheet-fed process where a shape or product is built with layers of metal foil without adding heat or melting the metal. Because the UAM process is unlike fusion-based AM processes, the UAM process does not melt the metal, so there is no change to the metal microstructure. This process allows the layering of multiple metals in one component. This aspect of UAM is highly advantageous for DTC cold plates.
The three components of one UAM weld head system are the “horn,” the transducers, and a set of brackets [Figure 1]. The horn is a highly engineered piece of tool steel designed to resonate at 20KHz. The transducers oscillate the horn with up to 9kW of power, and the brackets apply a downward force of up to 10K Newtons. During the build process, the horn is rolled over thin sheets of metal foil. As the horn is rolling over the sheet of metal (along the X-axis), the transducers oscillate the horn (on the Y-axis) while the brackets are applying a downward pressure (negative Z-axis). The oscillation “scrubs” the two metal layers together, dispersing the oxides and impurities. This action keeps the metals in a solid state while increasing their plasticity. The downforce allows the two metals, while plasticized, to bond together metallurgically. The most important result of this process is that the bond between the two metals (whether they are similar or dissimilar metals) is a full metallurgical bond with no intermetallic formation [8]. These pure metallic bonds mean there is no resistance to the transfer of electrical or thermal energy as expected from using a braze or solder. The net effect of this process is the true differentiator. A part will be one monolithic piece of metal formed from similar or dissimilar metals. Additional sustainability benefits to the UAM process will not be explored here. UAM provides thermal engineers with three distinct advantages:
– Freedom of Geometry
– Freedom of material
– Freedom of information
Freedom of Geometry
While UAM weld heads have been integrated into a variety of motion systems, a typical UAM system is installed into a CNC Mill. This configuration, called “Hybrid Manufacturing,” could be the breakthrough technology to increase DTC heat extraction. Combining UAM, an additive process, with a CNC mill, a subtractive process, allows for manufacturing complex interior channels, fins, turbulators, and evaporation geometry where every surface has a CNC surface finish and CNC accuracy [Figure 2]. Fusion-based 3D printers also allow freedom of geometry, but parts manufactured with that type of (AM) process have high surface roughness and wide dimensional variation. UAM allows designers the freedom to create the exact geometry they need at the exact location they require, with a CNC-quality finish.
Freedom of Material
UAM allows designers to put the metal they need where they need it. UAM does not rely on melting, so it avoids the complex metallurgical interactions of most welding processes. As a result, every layer in a UAM build-up (part) can be made from a different metal [Figure 3]. Many UAM heat exchangers are made with aluminum to reduce cost, and copper is printed in specific locations to wick heat away from a specific heat source. Similarly, materials with low coefficients of thermal expansion, such as Invar and Molybdenum, have been layered into heat exchangers to help mitigate CTE stresses between chips and metal cooling structures [Figure 4].
While copper has proven to be an excellent conductor (~400 W/m K) to extract heat from high-power electronics, many new higher-conductivity materials are emerging. UAM has the unique ability to embed one material inside another. That merges the properties of both materials to create a hybrid material designed by the engineer and made by the UAM system. Relevant to the construction of a DTC cold plate, various carbon-based materials could be embedded within the architecture of an aluminum or copper DTC cold plate. This material freedom could significantly enhance heat transmission from the device into the cooling liquid. Recent experiments with superconducting foils embedded into copper using a UAM process have shown great promise for creating high-performance laminates for electrical and thermal conductivity [9]. Simple combinations of aluminum, copper, and silver can be combined with UAM to extend the range of existing components. New materials such as graphene and superconductors embedded in DTC Cold plates could provide orders of magnitude higher heat flux.
Freedom of Data
The low-temperature nature of the UAM bond allows the embedding of sensitive electronics anywhere in a solid metal part. By embedding sensors directly into a heat exchanger, operators can get real-time in situ data for structural health monitoring, condition-based maintenance, or performance monitoring. In other industries, UAM has been used to embed fiber optic cables for sensing strain and heat. The sensors allowed the customer to measure strain and temperature throughout a complex engineering process. Other applications include embedding sensors in the wall of process piping for oil/gas and nuclear industries. Similar fiber optic sensor arrays would allow early detection of an overheated electronic device (GPU, CPU) within a DTC cold plate. Incorporating real-time data could play an integral role in a “smart” water flow manifold system that could direct the coldest inlet water to the server (or device) in real-time.
The design freedom that UAM provides has already been leveraged in high-consequence environments such as satellites. NASA significantly improved the performance of a satellite heat exchanger [10] [Figure 5] by over 30% with a simple UAM redesign. Similarly, UAM was employed to create a heat exchanger for Cube Satellites [11] for Utah State University with a combination of aluminum and copper. Similar concepts could be extended directly to DTC cold plates using other materials [Figure 6].
UAM can embed materials, devices, and sensors that have yet to be investigated in DTC cold plates. Accelerating heat extraction along with temperature sensing at critical points on and around the CPU/GPU, the server, and the rack, coupled with a software application that can leverage these sensors and embedded devices, could be the pathway to the next generation of DTC systems.
As chip density increases, server densities increase, and rack power and heat increase, air cooling may fall short of the cooling required with these AI/HPC data centers. DTC offers a robust option. The current architecture of the DTC cold plates has been greatly optimized. More innovative approaches should be investigated, emphasizing maximizing chip cooling by further developing current DTC cold plates. Given the applications mentioned above and the technology already employed that directly relates to a DTC form factor, it would hold that UAM could help.
References
[1] Neptune.AI “Deploying Large NLP Models: Infrastructure Cost Optimization, “Nilesh Barla, March 2023, https://neptune.ai/ blog/npl-modes-infrastructure-cost-optimization
[2] Data Center Frontier, Endeavored Business Media, LLC “Rack Density Keeps Rising at Enterprise Data Centers,” Rich Miller, April 2020, https://www.datacenterfrontier.com/cooling/article/11429037/rack-density-keeps-rising-at-enterprise-data-centers
[3] Research North America, H1 2023, North America Data Center Report, Jones Lang LaSalle I.P., Inc
[4] Holistic Study of Thermal Management in Direct Liquid Cooled Data Centres: from the Chip to the Environment, Mustafa Alaa Kadhim Kadhim, The University of Leeds, Institute of Thermo-Fluids, School of Mechanical Engineering, Apr 2018
[5] Data Center Knowledge, “Is Chip Cooling the Answer to Data Center Sustainability?” Soni Brown, Aug 2023
[6] Thermal Science, “Design and analysis of liquid cooling plates for different flow channel configurations,” Muhammad Farhan, Muhammad Amjad, Zia ulRehman Tahir, Zahid Anwar, Oct 2021
[7] https://gen3d.com/news-and-articles/the-seven-categories-of-am-processes/
[8] Comprehensive Review of Ultrasonic Additive Manufacturing, Adam Hehr, Ph.D, Mark Norfolk, Oct 2019
[9] 1 U.K. Atomic Energy Authority, Abingdon OX14 3DB, United Kingdom, 2 Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, United Kingdom, Development Of UAM For Rebco CC Terminations In High Purity Copper, Y. Dieudonne1†, A. Shchukin2, M. Burkhart3, M. Norfolk3, Fabrisonic LLC, Lewis Center, OH 43035, USA, † Corresponding author: yannik.dieudonne@ukaea.uk
[10] Maghsoudi, Elham, Mastropietro, A. J., Roberts, Scott, and Kinter, Bradley, “Experimental Thermal Performance Comparison of 3D Printed Aluminum Heat Exchangers vs Traditionally Manufactured Heat Exchangers,” Spacecraft Thermal Control Workshop (STCW), El Segundo, CA, March 2017
[11] Anderson, Lucas, Swenson, Charles, Mastropietro, A. J., Maghsoudi, Elham, Luong, Simon, Hofmann, Douglas, and Roberts, Scott, “Active CryoCubeSat Project: Design and Status,” Small Satellite Conference, Logan, UT, August 2017
