Tripling the Compute Capacity at the NASA Center for Climate Simulation

Overview

The NASA Center for Climate Simulation (NCCS) provides high-performance computing (HPC) and data storage services for NASA's Science Mission Directorate. As researchers increase the resolution of their weather and climate models, supercomputers and data storage systems must rapidly grow to support expanding computational requirements and the petabytes of data being generated.

Project Details

With a Scalable Compute Unit (SCU) design, the NCCS Discover supercomputer has been growing at a rate of approximately one SCU addition per year since 2006. In the fall of 2014, the NCCS embarked on an unprecedented expansion plan for Discover that would triple the cluster's computational capacity and double its online disk capacity. This effort also required the decommissioning and de-installation of five older SCUs and six aging storage systems.

Results and Impact

Beginning in late November 2014, the NCCS began installing three SCUs that have increased the overall size of the Discover cluster from 43,000 Intel Xeon processor cores performing 1.1 petaflops peak to 80,000 Xeon processor cores performing 3.5 petaflops peak. Staff installed these three SCUs in rapid succession over a period of only six months—the fastest and largest expansion ever executed at the NCCS:

• SCU10 – 30,240 Xeon Haswell cores, 1.2 petaflops peak (November 12, 2014).
• SCU11 – 17,136 Xeon Haswell cores, 713 teraflops peak (December 15, 2014).
• SCU12 – 17,136 Xeon Haswell cores, 713 teraflops peak (May 26, 2015).
• Total System – 80,000 Xeon (Sandy Bridge and Haswell) cores, 3.5 petaflops peak.

To handle the large amounts of data being produced by this compute expansion, the NCCS has also integrated an additional 20 petabytes of disk storage into the Discover environment, bringing the total capacity to 33 petabytes.

Why HPC Matters

As a result of this large system upgrade, the Global Modeling and Assimilation Office (GMAO) at NASA's Goddard Space Flight Center has been able to use 30,000 Haswell cores to run a global weather simulation at 1.5-kilometer resolution with more than 200 million grid cells. This represents the highest-resolution global simulation to date and the first ever to include interactive aerosols and carbon. The GMAO is leveraging these expanded NCCS resources to increase the resolution of their operational models and to examine features in the models that could not be resolved at lower resolutions.