NASA@SC18

Supercomputing

The More Things Change—The Less Sleep You Get

Overview

The Discover supercomputer cluster is the heart of the high-performance computing (HPC) services offered by the NASA Center for Climate Simulation (NCCS). The NCCS provides HPC and data storage services for NASA's Science Mission Directorate, including hundreds of NASA and university researchers. These users have come to rely on the regular technology upgrades to Discover to maximize their production of scientific results over the long term.

Project Details

Discover began in 2006 with 130 compute nodes containing 520 CPUs with a total capacity of 3.3 teraflops. Over the years, the NCCS has augmented Discover with new Scalable Compute Units (SCUs), including SCU14 in 2018. That upgrade brought the cluster to over 3,700 compute nodes and nearly 108,000 CPUs, with a total capacity of 5.1 petaflops.

SCU14 comprised the most change in a single upgrade that the cluster has seen. Many core elements were new to the NCCS, including the primary hardware vendor, CPU model, operating system, high-speed interconnect, and cooling system. Adding to the list of changes and challenges were new versions of software for switches, cluster management, and resource management.

Results and Impact

This year, the NCCS expanded the Discover cluster with an Intel Xeon Skylake-based SCU built with an Omni-Path high-speed network. This effort involved an operating system upgrade from SLES11 to SLES12 and multiple firmware and driver updates to support the new processors and interconnect. Working with new hardware vendors always brings challenges as new relationships are forged. This expansion was no exception, with numerous physical obstacles that had to be overcome. There were many lessons learned for both the NCCS and the multiple vendor partners involved, which will allow future expansions to proceed more smoothly.

Throughout this long and challenging process, the NCCS staff persevered and gained valuable experience with a new computing hardware vendor, many new hardware components, and with the updated software elements that comprise SCU14. Changing the list of MPIs that SCU14 supports versus the rest of the cluster added yet another layer of complexity. Ultimately, the time spent troubleshooting and learning about the multiple pieces of new technology has given the NCCS a broader and deeper understanding of the current HPC industry’s hardware and software landscape.

Why HPC Matters

With the latest upgrade to the Discover supercomputer, the NCCS continues to provide state-of-the art HPC resources to NASA High-End Computing (HEC) Program users in the form of faster, more efficient processors with one of the fastest high-speed interconnects available today. This upgrade, as with all previous upgrades, will enable advances in NASA science and engineering research that would not be possible without computational resources like Discover.

More Information

Bruce Pfaff, NASA Goddard Space Flight Center
bruce.pfaff@nasa.gov