NASA Blazes a Different Path to Energy-Efficient Supercomputing

Overview

For years, NASA had a very straightforward process for replacing high-performance computing hardware: over a three-year period, when it became more expensive to operate an older suite of hardware than it did to replace it with new products that could accomplish the same work, we simply replaced the old hardware. For NASA’s High-End Computing Capability (HECC) Project, that process changed when we reached the limits of our facility’s power, cooling, and floor-loading capacity, becoming a strategy of decommissioning the least productive hardware and replacing it with more capable counterparts. The impact was that we provided our users with less supercomputing capability than we would have without the limitations.

Additionally, with 25% of our total power consumption going to cool our systems and 50,000 gallons of water per day being evaporated, we wanted a solution that would expand our compute facility while being sensitive to the impact on our environment.

Project Details

The facility housing HECC systems is located at NASA’s Ames Research Center, near the southern end of the San Francisco Bay. The area’s temperate climate opened up solutions that are well suited to this location and allow for extremely energy-efficient approaches. Working with NASA Headquarters, which commissioned a study to evaluate possible expansion solutions, we determined the best path forward was to deploy a prototype module system with a goal of reducing both power and water utilization. Using 25% of total power consumption for cooling equates to a 1.33 Power Usage Effectiveness (PUE) for the primary NASA Advanced Supercomputing (NAS) facility; our goal was to deploy a system with a 1.06 PUE and to all but eliminate water utilization.

We deployed the first prototype module in late 2016, and it has been operational for about a year. It houses a 1.2-petaflop system, called Electra, that uses about 400 kilowatts (kW) of power. The module exceeded our optimistic goals. Over the third quarter of 2017, the PUE for this system was 1.025, beating our target of 1.06; water usage for the entire year was about 55,000 gallons, a reduction of more than 97% over the primary facility.

We successfully used outside air to cool the compute system throughout the year, only running water for cooling when needed. The outside air is pulled into the module through large particle filters and enters a mixing chamber, where it can be mixed with air from the module’s hot aisle to lower the humidity or raise the temperature. It then passes through small particle filters and a mesh media that can contain water to cool the air (if needed) before passing into the cold aisle and through the computers.

Results and Impact

The success of the first prototype led to the acquisition of a second, highlighting another advantage of this approach: expansion on demand. By this time, new technologies became available that were not an option with the first prototype. The second module can hold twice the number of nodes, and can remove more heat load—allowing us to increase total power from 500 kW to 1.2 megawatts. The new system uses the same air/water cooling technology as the first, but also has direct-to-chip heat transfer to a cold-water loop. Early testing is in line with the first module with a PUE value of 1.03, exceeding our goal of 1.08 for the new system. When fully populated, the new Electra system—comprising the nodes in both modules—will reach a peak performance of over 8 petaflops.

What’s Next

Next year, we will begin construction on a new modular facility based on the technologies proven in the prototypes. This new facility will enable additional expansion to meet NASA’s future challenges.

More Information

• See the poster for this demonstration (PDF, 3.5MB), as shown in the NASA booth at SC17.