By Yogesh Fulpagare and Atul Bhargav
Energy Systems Research Laboratory, Indian Institute of Technology Gandhinagar, India
With the advancement of technology, the access to information has become rapid as well as ubiquitous. These mammoth resources which can be accessed through one click are stored and processed in data centers. These data centers are one of the largest and fastest growing energy consumption systems, and are expected to consume more than 140 billion kilowatt-hours of energy by the end of this decade [1]. Increasingly compact processors have shrunk the server volumes and increased the heat management issues faced by the data centers. Hence sufficient number of investigations over the last decade has focused on improving the thermal performance of data centers. Researchers have used numerical, experimental and computational fluid dynamic (CFD) analyses to study data centers from macro to micro [2] level with an aim to minimize the energy requirement for cooling. However there are still many challenges to be overcome in order to achieve efficient operation of data center due to its dynamic behaviour. Such challenges can be categorised in terms of: (i) the raised floor plenum data center models, (ii) rack layout with thermal analysis, (iii) energy efficiency with performance metrics, (iv) data center dynamics (control & life cycle analysis), (v) data center model validation and (vi) programming based optimization [3]. Coupling CFD models with real time control and job placement algorithms can provide valuable insights and solutions to the above mentioned concerns. Although air cooling in data centres in some instances may reach its limits due to high thermal loading, it can still be managed effectively through close examination of flow and thermal dynamics of the system. Data center systems can be parameterised through standard metrics such as Power Usage Effectiveness (PUE) and Energy Reuse Effectiveness (ERE) through the server inlet-outlet temperatures, cold air flow rates, and power consumption. Hence there is a need for standardisation of these variables using the knowledge of system thermal behaviour through real-time measurements. Thermal behaviour of the system can be efficiently captured through predictive modelling on real-time basis and can be integrated with the control strategies.
The focus of this article is to consider how further improvements in efficiency of data centers can be obtained. In this context, a multi-disciplinary approach is proposed to data center cooling system design, data center layout design and data center operation. In previous work [4] the authors developed CFD models to understand the dynamics of the data center system which are the first building block for further model development. These CFD models leads to conclusion that the data center subsystems such as plenum obstructions, perforation of tiles, server fans, structural arrangement of server racks and cooling systems all affects data center system performance. A summary of key challenges faced by data center personnel in the field of thermal management can be given as follows:
- Are these steady state and transient CFD models sufficient to model data center thermal issues? Is steady state CFD analysis sufficient? What is about the computational time?
- If the answer to the previous question is yes, then what is the accuracy of these models? If no, then what are the tools apart from CFD will help to address the heat management issue?
- Data centers are highly dynamic in nature, is it possible to develop the models that can capture these dynamism?
- Is there standardization to tackle the heat management through ASHRAE? Are those helpful for data center operators?
While addressing solutions to the above questions it’s intuitive that the structural arrangement of the data center system may affect the system performance. To cool the servers efficiently, data centers have special arrangements such as raised floor plenum (RFP) with different supply and return structure though overhead block arrangement. These arrangements can substantially alter the data center thermal profile.
To understand the thermal performance of the data center system, detailed tile perforation of data center system was studied from the previous work [5]. At the same time perforation of the tiles and obstruction in the plenum chamber affects thermal profile of the data center system. Hence, using the analysis of perforation tile details the flow distribution and thermal performance of the system was studied and validated for various test cases on conventional raised floor data center. Thus there are two prominent cases reviewed in this article, 1) Verification of detailed perforated tile geometry and quantification of tile air flow-rate and 2) Flow distribution analysis in the raised floor plenum chamber with seven test models.
To quantify the effect on cold air flow rates by different perforation of tiles a rectangular computational domain was prepared for CFD study. Each tile of 2m × 2m at three corners of the domain modelled with 25%, 36% & 50% perforation. The grey area is flow domain while the red area indicates the perforation in figure 1. The geometry of the tiles was analysed in terms of flow rates and thermal profiles at different perforations. While comparing the perforated tile modelling with the available actual system data it was found that to model 25% and 56% tiles, the CFD model must incorporate 56% & 100% open area, respectively. The reason behind it can be explained in terms of momentum loss of air. Cold air from Computer Room Air-Conditioning (CRAC) units while passing through perforated tiles forms a single large jet through combinations of small jets of air formed due to perforation. During this phenomenon cold air loses significant momentum.
Furthermore seven computational models were built with various combinations of plenum chamber with and without obstructions, data center room and server racks (Table 1). All the computational models were incorporated in a commercial CFD software [6] for steady state analysis. The operating characteristics of the system are detailed in Table 2.
Table 1. Porous inertial resistance (P) and number of mesh cells for each CFD model assessed.
| Model | Geometry | P (kg.m-4) | Number of Mesh cells in 100,000 | |
| A | Plenum chamber | 29 | 1.49 | |
| B | Plenum chamber with pipes | 29 | 63.4 | |
| C | Plenum chamber (with modified pressure loss coefficient) | 630 | 0.9 | |
| D | Plenum chamber with pipes (with modified pressure loss coefficient) | 630 | 6.34 | |
| E | Plenum chamber having pipes above a data center room | 630 | 0.57 | |
| F | Plenum chamber without pipes above a data center room containing server racks | 630 | 0.94 | |
| G | Plenum chamber with pipes above a data center room containing server racks | 630 | 1.04 | |
Table 2. Data center operating conditions.
| Cold air temperature | 15 °C |
| Ambient temperature | 25 °C |
| Cold air flow rate | 6 kg/s (per CRAC) |
| Server rack heat source | 10 kW (per rack) |
To overcome meshing challenges while introducing obstructions in plenum chamber all the pipes were modelled with rectangular cross section for ease in polyhedral mesh generation. Figure 2 shows the structure of the obstruction incorporated inside the plenum chamber with meshing details.
Figure 2. Details of plenum obstructions incorporated in all seven computational domains, Table 1.
The details of Navier-Stokes mathematical formulation with turbulence models can be referred from [4]. The air flow inside the system was assumed to be steady state and isothermal. Porous resistances across the tiles plays an important role in modelling perforated tiles in computational domain. The momentum equations with segregated solvers includes the porous media body force where the inertial and viscous resistances across the tiles can be provided. These resistances are sensitive with pressure loss coefficients. Hence first four models (A to D, Table 1) were useful for fixing the pressure loss coefficient to account the momentum loss for its further use in next complex models.
Tile cold air flow-rates with modified pressure loss coefficients were matched with the experimental data from the literature [7]. Hence, it is suggested that the modified pressure loss coefficient values should be used while modelling the perforation tiles. The flow field and thermal profiles was observed on the models (F & G, Table 1) that contains all details of the data center system (Figure 3). It was found that the obstruction can lead to an increase in temperature by 2.5 °C as well as decrease the air flow rate up to 80% [4]. The performance of the data center can be enhanced by routing the under floor blockages, ducts & pipes. Therefore there is need to develop standardised codes that would guide the placement of obstruction for efficient construction of the system.
| (a) | (b) |
Figure 3. Temperature tracking through streamlines in a CFD model of data center room with (a) plenum and servers without obstructions (Model F) and (b) with obstructions (Model G), Table 1.
There are many such arrangements of the racks that may lead to efficient thermal management of the data center. Such different structural arrangements can be assessed using CFD analysis but will be difficult to validate. Still the question of CFD analysis remains questionable in terms of computational time and experimental validation. The steady state CFD modelling may be sufficient to understand the physics of the actual data center. As a possible improvement, transient analysis has been reported in recent studies but are expensive in terms of computational time [8], [9]. Also there are many difficulties while actual modelling of the server systems, cooling units and perforation tiles which is the focus for most researchers including ASHRAE standards. However the real time data center system data of various variables will be a key for CFD model validation as well as formulation of the mathematical models of the system. Data center itself is very complex and dynamic in nature and hence difficult to have the data with all possible input changes of the system. Hence there is a need for a scaled down testing facility which can mimic the actual data center facility. However, the implementation of such scaled down testing is difficult with flow and thermal dynamics matching between actual and scaled down system needing to be captured through dimensionless analysis.
The successful implementation of prototype testing will be breakthrough research for data center operators. Apart from that, all the CFD analysis and system testing will provide real-time data of server temperature, air flow rates, etc. Controlling the air flow rate as per the demand in specific server rack locations is the main objective for all type of data centers to save the power consumption. This objective forms the basis to develop the control strategies that includes the thermal model from the available testing data.
Acknowledgement
Yogesh Fulpagare received funding from Ministry of Human Resource Development, Government of India and Ministry of Science & Technology (Grant number DST/INSPIRE Fellowship/2013/451).
References
[1] R. Brown, E. Masanet, B. Nordman, A. Shehabi, J. Stanley, D. Sartor, and P. Chan, “Report to Congress on Server and Data Center Energy Efficiency : Public Law 109-431,” U.S. Environmental Protection Agency, ENERGY STAR Program.
[2] Y. Joshi and P. Kumar, Energy Efficient Thermal Management of Data Centers. New York: Springer, 2012.
[3] Y. Fulpagare and A. Bhargav, “Advances in Data Center Thermal Management,” Renew. Sustain. Energy Rev., vol. 43, pp. 981–996, Mar. 2015.
[4] Y. Fulpagare, G. Mahamuni, and A. Bhargav, “Effect of Plenum Chamber Obstructions on Data Center Performance,” Appl. Therm. Eng., vol. 80, pp. 187–195, Apr. 2015.
[5] W. A. Abdelmaksoud, H. E. Khalifa, T. Q. Dang, and B. Elhadidi, “Experimental and Computational Study of Perforated Floor Tile in Data Centers,” 12th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), pp. 1–10, 2010.
[6] STAR CCM+, http://www.cd-adapco.com/products/star-ccm%C2%AE
[7] E. Samadiani, J. Rambo, and Y. Joshi, “Numerical Modeling of Perforated Tile Flow Distribution in a Raised-Floor Data Center,” J. Electron. Packag., vol. 132, no. 2, p. 021002, 2010.
[8] G. Varsamopoulos, M. Jonas, J. Ferguson, J. Banerjee, and S. K. S. Gupta, “Using transient thermal models to predict cyberphysical phenomena in data centers,” Sustain. Comput. Informatics Syst., vol. 3, no. 3, pp. 132–147, Sep. 2013.
[9] M. Jonas, R. R. Gilbert, J. Ferguson, G. Varsamopoulos, and S. K. S. Gupta, “A transient model for data center thermal prediction,” 2012 Int. Green Comput. Conf., pp. 1–10, Jun. 2012.
Contact Information
Yogesh Fulpagare
Indian Institute of Technology Gandhinagar,
VGEC Complex, Chandkheda,
Ahmedabad – 382424, Gujarat, India
Email: fulpagare_yogesh@iitgn.ac.in
Phone: +91 972 762 1699
