Liquid Cooling for Datacom Equipment Centers

Introduction

The increasing heat load densities in datacom equipment centers require ever more sophisticated approaches to cooling. Air-cooled systems are now struggling to provide the needed level of thermal performance for many installations and energy efficiency is an increasing concern. In fact, the standard implementation of air-cooled designs results in cooling energy costs that, over the typical three year life of the equipment, can actually exceed the cost of the IT equipment itself [1, 2]. In response to all of these issues, many datacom equipment center managers are considering and/or implementing liquid-based systems to manage the cooling needs of high power density areas.

Recognizing this shift, the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) Technical Committee 9.9, Mission Critical Facilities, Technology Spaces and Electronic Equipment was formed in 2001 [3] and recently published guidelines for liquid cooled datacom equipment centers [4]. This concise, 100-page book is the result of a combined effort by experts from the thermal management, IT equipment, facilities design, and user communities. While it serves as a resource for datacom equipment centers, it also serves as a useful reference for liquid cooling design.

This article provides a brief review of the various liquid cooling options for datacom equipment centers and a summary of the overall design implementations, from buildings’ facility cooling systems down to the IT rack/cabinet. Also addressed are considerations for diagnostic monitoring, reliability, and backup redundancy. For more detailed information, consult the new ASHRAE guidelines [4].

Definition of Liquid Cooling

A liquid-cooled system dramatically differentiates itself from an air-cooled system in the datacom equipment center itself. Liquid cooling is defined as the case where liquid must be supplied to an entity for operation. Examples of liquid cooled systems include liquid-cooled racks, datacom equipment, and electronics. In each of these definitions, a single or two-phase coolant system may be employed. Additionally, a variety of coolants, including water, ethylene glycol/water solution, refrigerants, or high dielectric fluid, can be used in liquid cooling implementations. Air cooling describes the case where only air is supplied to an entity for operation. When liquids are employed within separate cooling loops that do not communicate thermally, the system is considered air cooling. According to this definition, chilled water computer room air conditioners and heat pipes are two examples of air-cooled devices.

Datacom Equipment Cooling Overview

A datacom cooling system involves many steps, from removing heat from electronic components such as microprocessors to eventually exhausting that heat to the environment outside the building. Figure 1 illustrates the multiple liquid loops and associated heat exchangers in typical datacom cooling systems. The building facilities part of the system (including the Condenser Water System [CWS] and the Chilled Water System [CHWS]) may be shared with other cooling loads within the building or it may be dedicated solely to the datacom equipment. The significant design details of the CWS are dictated by the usual building control considerations, such as peak load, outdoor environmental conditions, redundancy, and required flexibility for expansion. A CHWS usually provides 50�F (10�C) water to the datacom equipment center.

Figure 1. Overview of a typical liquid-cooled design for facilities and datacom equipment.

The Cooling Distribution Unit (CDU) transfers heat from the Technology Cooling System (TCS) to the CHWS and typically delivers coolant to several IT racks or cabinets. There are multiple benefits to implementing a CDU and TCS including:

• Preventing condensation by delivering coolant to the rack, equipment, or electronics above the dew point
• Isolating the electronics from the harsher facility water in the CHWS or CWS loop and minimizing the coolant volume near the technology so that a coolant leak would be less catastrophic
• Providing flexibility on the coolant as well as its temperature, pressure, and purity since the CHWS can only provide one condition of coolant fluid flow (temperature, pressure, and purity)

 

The system may or may not include a Datacom Equipment Cooling System (DECS).

While Figure 1 depicts a generic system overview, alternative implementations can be considered. For example, one common alternative buries a dedicated CDU in each cabinet and essentially eliminates the TCS loop. In this case, more but smaller CDU’s and a more complicated CHWS functionally replace the central CDU and TCS loops. Such trade-offs can be optimized for each application depending on the specific requirements and cost.

Figure 2 shows four commonly implemented approaches to air- and liquid-cooled cabinets. (Reference 4 provides more configurations and their merits.) As opposed to the purely air-cooled cabinet, the liquid-cooled designs enable higher densities. Using cold plates, heat sinks, liquid heat exchangers, and air heat exchangers, many hybrid combinations of air- and liquid-cooled designs can be tailored for specific applications.

Figure 2. Four methods for cooling datacom racks or cabinets.

Common implementations today are the open air-cooled with air/liquid heat exchanger and the closed air-cooled datacom equipment in a liquid-cooled cabinet. Figure 3 shows such a hybrid arrangement where a closed air- and liquid-cooled cabinet uses internal air circulation for general cooling supplemented by liquid cooling of targeted components with high dissipation requirements.

Figure 3. Closed liquid-cooled cabinet with air-cooled datacom equipment.

Liquid cooling reduces energy consumption by removing heat at the sources and/or conditioning a small controlled environment versus conditioning the entire room. It makes sense to apply your cooling watts only where they are needed and liquid-cooled methods allow that concept to be implemented more effectively.

Design Guidelines for the Liquid Cooling Infrastructure

The new ASHRAE book [4] provides guidelines on the interface requirements between the CHWS and the TCS, and also outlines the requirements of those liquid-cooled systems that attach to the datacom electronics rack.

First, the basic system design requirements are addressed. For the TCS loop, the minimum liquid supply temperature must remain above the dew point at all times to prevent condensation. The simplest method is to set a fixed control point above 63�F (17�C) � the maximum dew point specification for a Class 1 environment. (See Thermal Guidelines for Data Processing Environments [5].)

A more sophisticated approach controls the temperature at a fixed offset above the continuously measured dew point. Liquid temperature control capability is typically present in both the chiller and the CDU. Another basic requirement is the maximum liquid pressure for both the TCS and the DECS loop, which for water should be no greater than 100 psig (690 kPa). Well designed chillers and CDU’s will already have the necessary sensor and control elements for maintaining the temperature and pressure of the liquid in their loops. With these parameters set, the design approach is driven by water quality, total cooling capacity, and individual requirements for each rack/cabinet.

Table 1. Recommended Water Purity Limits

For water-cooled CHWS and TCS loops (including those with glycol additives), the liquid purity level is a critical factor in determining the system’s performance and reliability. Recommended limit values of the major contaminants for the liquid treatment program have been included in the ASHRAE guidelines and are reproduced in part in Table 1.

Since purity of the TCS loop is much more stringent than that of the CHWS or CWS loops, additional design considerations are necessary to extend the lifetime and minimize maintenance. Design recommendations include the following:

• Size piping for water flow rates and maximum velocity. (See [4], pp. 74-75.)
• Incorporate strainers and/or side-stream particle filters to remove suspended material.
• Avoid dead-legs or sub-loops, which may be isolated for extended periods of time.
• Use only appropriate wetted materials such as copper, brass with less than 15% zinc, stainless steel (304 or 316), or ethylene propylene diene monomer (EPDM) polymer that has been peroxide cured.
• Specify corrosion inhibitors and biocides suitable for the materials (avoid oxidizing biocides; e.g., chlorine and bromine based).
• Do not use both copper and aluminum in the same system, as corrosion inhibitors are generally different.
• Specify the makeup water source and its requirements (deionized, distilled, or reverse osmosis).

 

Designing for Reliability

Designing a liquid cooled system for reliability encompasses several aspects, from component selection and sizing to enabling effective monitoring and maintenance. Critical components that tend to dictate the overall system reliability include pumps, fans, blowers, electronic controllers, heat exchangers, and sensors. However, even passive elements of the system such as fittings, tubing, fasteners, and electrical connectors can cause reliability problems if not selected and applied appropriately. Availability (or uptime) is dependent on the component reliability but can be further improved by designed redundancies and back-up subsystems.

These techniques for maintaining effective operation are especially relevant in cases of single fault events, power outages/disruptions, water supply disruptions, and operator error. Provisions for effective monitoring and trouble-shooting of the TCS allow operators and maintenance technicians to keep the cooling system running efficiently and reliably. This is implemented most effectively with a direct interface to the Enterprise Network (IT) and/or infrastructure facilities network. Monitored signals and alarms must be determined for the overall system in conjunction with the individual component and equipment manufacturers’ recommendations for specified range limits, alert values, and alarm values.

Summary

The datacom equipment center trend is to pack more and more “IT power” into the same space. While this IT power is necessary for the data processing function, it means more “real power” (in the form of kWatts of dissipated heat) must be effectively managed. Today’s data centers are replacing and enhancing traditional cooling implementations that rely on inefficient air-cooled racks with liquid-cooled approaches. The primary benefits delivered by an effectively designed liquid cooling approach are:

• Provides high capacity heat flux density (i.e., removes more heat from the same space)
• Increases IT equipment reliability by enabling lower temperature operation of heat generating electronic components (e.g., microprocessors and power supplies)
• Reduces energy cost with improved efficiency

 

Liquid cooling is not simply a passing fad since its advantages have become more compelling with each generation of IT equipment. As a result, facility managers, data center managers, and thermal engineers are recognizing the merits of liquid cooling and there is a growing trend in its application to the IT environment.

ASHRAE’s Technical Committee 9.9 mission is recognized by all areas of the datacom industry as the unbiased engineering leader in HVAC and an effective provider of technical information. The committee has written the datacom series of books in support of this mission. For more details on the books, see http://tc99.ashraetcs.org/content.html.

References

1. Belady, C., “In the Data Center, Power and Cooling Costs More than the IT Equipment It Supports,” ElectronicsCooling, Vol. 13, No 1, February 2007.
2. Brill, K., “The Invisible Crisis in the Data Center: The Economic Meltdown of Moore’s Law,” Uptime Institute, available at http://uptimeinstitute.org/wp_pdf/(TUI3008)Moore’sLawWP_080107.pdf
3. Schmidt, R., “ASHRAE Committee Formed to Establish Thermal Guidelines for Datacom Facilities,” ElectronicsCooling, Vol. 11, No. 1, February 2005.
4. “Liquid Cooling Guidelines for Datacom Equipment Centers,” American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., 2006.
5. “Thermal Guidelines for Data Processing Environments,” American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., 2004.