Simulation Helps Meet Thermal Challenges of World’s Fastest Supercomputer

IBM���s Roadrunner will be capable of a sustained speed of up to 1,000 trillion calculations per second or one petaflop. Roadrunner will utilize 16,000 Cell B. E. processors, which were originally designed for Sony Playstation video game platforms. The supercomputer will also include 16,000 AMD Opteron processor cores. The Roadrunner will primarily be used for simulations to ensure the safety and reliability of the nation’s nuclear weapons stockpile. The Roadrunner occupies only a portion of the Metropolis Center at the Department of Energy’s (DOE’s) Los Alamos National Laboratory (LANL), which is filled with other high performance computing equipment.

One of the key challenges of owning and operating this enormous computing resource is addressing the critical thermal management challenges. A conventional data center is composed of racks that typically dissipate between 2 and 6 kiloWatts (kW). The high density racks in the Metropolis Center, on the other hand, dissipate up to 20 kW per rack. LANL first contracted with CH2M HILL several years ago to perform thermal simulation as part of planning for the installation of Roadrunner as well as other expansion projects in the Center.

“We use Flovent as our thermal simulation tool at CH2M HILL because it provides many features that simplify the task of modeling data centers,” Solberg said. “Flomerics also provides a team of support engineers that are experienced in addressing data center cooling issues.”

In September 2006 CH2M HILL modeled the west side of the Metropolis Center in order to evaluate the effect of adding the Roadrunner Phase 1 and Phase 2 Equipment. In 2007 CH2M HILL expanded the model to include the entire Center, including Roadrunner Phase 3 equipment and Tri-Lab capacity clusters Scalable Units (SUs). The SUs are scalable high performance clusters used for a variety of research performed at the three sister national laboratories Lawrence Livermore National Laboratory (LLNL), Sandia National Laboratory, and Los Alamos National Laboratory

“The simulation demonstrated that the heat loads involved in the operation of this data center were managed well by the existing Metropolis Center building design,” Keith Kibbee, also a CH2M HILL engineer, concluded. “It showed that the data center could handle all of the equipment that had been planned and projected while adding 14 computer room air conditioners.” Furthermore, the engineering team used Flovent to test alternatives that optimize the use of the available cooling capacity before ever having to touch a perforated floor tile in the actual facility.

CH2M HILL engineers also developed several proposals to increase the cooling efficiency of the data center so it could be able to handle even greater loads. “We noted that the current facility has no physical barriers between hot aisles and cold aisles other than the server rack itself,” Solberg said. “Theoretically these changes could bring the cooling efficiency of the Metropolis Center close to 100% and allow the thermal load in the data center to be increased by approximately 20% without adding cooling capacity. This application demonstrates how simulation can save money in the data center by more efficiently utilizing available cooling resources.”