Data center cooling used to be a simple affair. You put a couple of computer room AC (CRAC) units against the wall, hook them up to the building chiller system and blast cold air under the floor. By having the racks arranged in alternate hot aisles and cold aisles with perforated tiles on the cold aisle floor, the cool air is sent up to the front of the racks to keep them cool.

Try telling that to Jan Petersen, IT production manager at Danfoss A/S, a worldwide manufacturer of refrigeration and valve products based in Denmark. His server room has a couple of large concrete beams running in parallel along the ceiling. These beams mess up the air flow dynamics of the space, creating air pockets that run the length of the room and trap about four feet of hot air. The top units in the racks were reaching temperatures of 25 to 29 degrees C at the front and were outputting air at 40 degrees C.

"The top servers of our old racks were often overheating," says Petersen. "It was very difficult to get cold air to the top of the rack."

Such problems are commonplace these days. Thus IT managers are faced with significant cooling problems. In response, they are supplementing (and sometimes replacing) the time-honored CRAC and hot aisle/cold aisle system with in rack cooling units of various configurations.

"Cooling and power are eclipsing space and availability as the main concern of data center managers," says Andreas Antonopoulos, an analyst at New York-based IT research firm Nemertes. "This trend is likely to get worse in the coming years as demand pushes up energy prices.

Changing Priorities

A survey of the members of the Data Center Users Group of power/cooling vendor Liebert Corp revealed that heat/power density has displaced availability as the top facility concern. 33 percent admitted that they will be out of power/cooling capacity by 2007. 96 percent stated they will be out of power/cooling capacity by 2011.

Part of the reason for this is far greater deployment of blades. 70 percent of the Data Center Users Group are implementing blades or have them in place. As a result, 55 percent have more than 8 kW in their most heat dense rack; and 20 percent have more than 20 kW in at least one rack.

Data centers were traditionally architected for the power consumption and cooling of 2 to 3 kW per rack. Yet the user survey showed that only 2 percent have racks remaining with less than 2 kW.

Take the case of a simple 1U AMD Opteron or Intel Xeon server. Each consumes approximately 300 - 400 watts. A rack of 24 machines, then, reaches somewhere between 7.2 and 9.6 kW.

"Once you get above 100 watts per square foot or about 6 kW per rack, under-floor cooling starts to become inadequate," says Michael Bell, an analyst at Gartner Group.

Such numbers are quite common these days. A Dell PowerEdge rack can pack 60 blade servers in a 42U enclosure equates to 30.4 kW per rack. IBM and HP blade racks record similar or even higher scores.

At Danfoss, for instance, the data center used to consist of Compaq servers that consumed no more than three kW per rack. Three CRAC at the perimeter produced enough cold air to keep temperatures down. But some parts of the room suffered due to poor air flow. The upper reaches of the racks suffered badly and failures resulted.

"We had some parts of the room with plenty of cold air, and other parts with very little," says Petersen. "We even had negative pressure in some cold aisles i.e. the air was being sucked back down into the floor rather than fed to the front of the servers."

The situation worsened considerably over the past three years as IT introduced a SAN which included several EMC and HDS disk arrays, a large StorageTek robotic tape library, and expanded its number of servers to around 1400 servers - split between two facilities situated about half a mile apart. Most recently, it has been mainly adding HP DL 380 rack servers and HP blades.

These acquisitions, of course, brought cooling demands through the roof. Danfoss implemented an InfraStruXure system by APC-MGE of West Kingston, RI. This cooling system provides complete hot aisle containment i.e. no hot air from two enclosed racks is allowed to spill out into the room. All HP blades and recent rack servers are positioned within the containment cube. Inside are several APC InRow RC units that utilized chilled water which is piped in under floor.

"Our servers are no longer overheating," says Petersen. "When we add a rack of blades, we always add two InRow RC units - one on each side."

23 racks of HP rack servers each require about 4 to 6 kW per rack. 5 racks of HP blades, on the other hand, consume 10 to 15 kW each. They are cooled by 11 APC RC cooling units. Due to their heat density, an RC unit is placed on each side of every blade enclosure. In addition, two APC power racks are required within the cube for power input and distribution. The temperature inside the cube is maintained at 27 degrees C. Outside air is 21 degrees C.

"Two years ago, we received overheating alerts quite often," says Petersen. "Now we have control over the environment. We could never have put blades into the old environment due to their heat output."

Water versus Refrigerant

Not everyone, though, wants water being pumped into the data center. Chilled water running right beside racks of expensive hardware is regarded as risky in some circles.

"Many people are worried about putting water in their data centers," says Antonopoulos. "Some of those concerns are unjustified as well-engineered systems would not really increase the risk of flooding."

As an alternative to water, refrigerant can be used. As it evaporates at room temperature, any leaks don't result in flooding. Egenera, for example, has partnered with Liebert to improve the cooling capabilities of its blades. Known as CoolFrame, this system integrates Liebert XD cooling technology with the Egenera BladeFrame system.

Liebert XD is a waterless cooling solution that includes a pumping unit or chiller and an overhead piping system to connect cooling modules to the infrastructure. One pumping unit or chiller provides 160 kW liquid cooling capacity for up to eight BladeFrame systems. They are mounted directly to the back of the BladeFrame. It adds $300 to $400 per blade to the BladeFrame price tag.

"Liebert XD uses a liquid refrigerant to handle up to 20,000 watts per rack and is highly effective," says Bell.

The system doesn't have to be bought with Egenera blades, however, Liebert also sells it directly. This above the rack refrigerant system is known by the broad term of the XD Adaptive Cooling platform. It delivers 5 kW to 20 kW per rack. Ceiling mounted cooling units (XDO) above the hot aisle and rack mounted units (XDV) can be used as part of this approach. The company also offers XDH in-row units that placed between racks if there is a lack of overhead space for the XDO.

Pomona Valley Hospital Medical Center (PVHMC), for example, had to cope with greatly increased heat loads. CIO Kent Hoyos investigated chilled water solutions, but decided not to risk bringing water into the room. He called in a local engineering firm to help with data center redesign. They recommended Liebert XD high-density cooling.

The coolant is pumped as a liquid from above into the cooling module. There, it vaporizes to a gas when it absorbs heat energy.

"I liked the fact that there is no risk of liquid leaking into my data center," says Hoyos.

He reports that when all of his 20 XD units were running at full capacity, the temperature in the data center fell by more than 30 degrees. The system eliminated heat-related equipment failures and even reduced help desk calls. PVHMC expects that it will be able to double its present server capacity using existing cooling equipment.

Cooling Alternatives

The newest product in the Liebert arsenal is the XDFN. This is a refrigerant-based or chilled-water system that can chill 10 to 30 kW per rack in a cabinet that encloses several racks and includes built-in cooling. Although it is typically used with overhead piping, under-floor pipes can also be used.

Similarly, APC-MGE offers the InfraStruXure InRow RP, available in both chilled water and refrigerant-based designs, though the company appears to favor water overall. Instead of constant speed fans, a system of sensors monitors temperature and ramps up or down fan speed and water pressure in order to reduce operating costs while improving effectiveness. An integrated humidifier provides room-level moisture control. The RP can cool up to 70 kW per rack (water) or 37 kW (refrigerant).

But Liebert and APC-MGE are not the only game in town. Rittal GmbH of Germany, while known more as a rack manufacturer, also sells cooling units. These are vertical models known as the liquid cooling package (LCP). HP uses a modified version of the LCP and rebrands it as the HP Modular Cooling System.

The LCP provides up to 20 kW per rack. It can be purchased as a modular system with about 6 kW of cooling per module. Water is normally pumped from underneath the floor. If the end user has an existing chiller system, they hook the system up to it. Otherwise, Rittal offers a chiller unit.

Rittal's LCP Plus, on the other hand, pushes the cooling potential up to 40 kW per rack if a 51U system is installed. A standard 42U LCP Plus gives up to 28 kW.

Knurr AG of Germany is another cooling vendor that designs units for high-density environments. At the low end of its product range is the 3U CoolServe. It slides right into traditional racks for supplemental in-rack cooling.

The company also offers several other models: CoolAdd, a door that is connected to the rack server with three large fans and a liquid cooling system inside the door. It is designed for up to 8 KW of power in the rack; the Miracel rack does 4.5 to 7 kW cooling via a vertical unit that sits beside the rack; and CoolTherm, a closed cabinet application i.e. an integrated rack with cooling included.

And then there are more exotic approaches such as SprayCool by ISR Inc. of Liberty Lake, WA. This is a system that cools processors directly by spraying a chemical agent onto the chip set.

Imminent Redesign

According to Gartner Inc. of Stamford, CT, half of all data centers will need to overhaul their cooling solutions within the next couple of years. At the same time, the number of professionals qualified to design and run modern data centers will decline by 45 percent per AFCOM, an Orange, CA, based association of data center professionals. Thus expect to see more and more data centers and server rooms being completely reorganized. They will begin turning to easy to operate and more efficient cooling technology in order to cope with rising server densities.

There are many strategies, of course, to address growing cooling management. But Gartner's Bell believes that water has to be seriously considered as a technology to adopt. Even if a data center isn't quite ready to adopt liquid cooling, he believes they better start thinking ahead. While that doesn't necessarily mean installing pipes to every server, he suggests that companies at least make sure they have the plumbing infrastructure in place to make water cooling easy to implement in the future.

For those ready to begin, he says they should start with in-row cooling for their highest-density servers. If they water cool those and make the system work well there, they will have learned enough to then effectively expand the system throughout the facility.

"Although the water is still somewhat experimental, it is ready to penetrate the market in a big way in next two or three years," says Bell. "As water is 3500 times more efficient than air at removing heat, it will become the preferred method of cooling."

Drew Robb is a freelance writer specializing in IT.

 

Forward this page to a friend

Your Email: *

Your Name: *

Send To: *

Enter multiple addresses on separate lines or separate them with commas.

Message Subject: (Your Name) has forwarded a page to you from NaSPA.com

Message Body: (Your Name) thought you would like to see this page from the NaSPA web site.

Your Personal Message: *

Your vote:

1

2

3

4

5

Average: 4 (1 vote)