NASA@SC18

Supercomputing

HPC in the Cloud? … Yes, HPC in the Cloud!

Overview

For its cloud computing strategy, the NASA Center for Climate Simulation (NCCS) is taking a portfolio approach extending on-premises assets to commercial clouds. An evolving capability promotes easy movement of applications and data among “cloud” systems ranging from traditional, very disciplined high-performance computing (HPC) clusters to customizable virtual environments. The NCCS loosens the normal definition of cloud to include HPC clusters because, at its core, cloud computing is about shared assets—something a batch-driven cluster does very well.

The NCCS HPC cluster is the 108,000-core Discover supercomputer, an assembly of Linux scalable capable of over 5 petaflops, or 5 quadrillion floating-point operations per second. Discover is particularly suited for large, complex, communications-intensive problems employing large matrices and science applications.

As combinations of remote sensing observations and model outputs have grown, NASA scientists are increasingly burdened with both the necessity and complexity of large-scale data analysis. This challenge leads many users to apply traditional HPC solutions like Discover to solve their big data problems. While this approach limits data movement, HPC systems are optimized for large-scale simulations and not for data analysis.

To address these issues and alleviate some of the strain on Discover, the NCCS created the Advanced Data Analytics Platform (ADAPT). ADAPT combines HPC and virtualization technologies to create an on-site private cloud optimized for large-scale data analytics. This system is used by a variety of NASA research projects, including the High Mountain Asia Team (HiMAT). ADAPT’s success led directly to the Goddard Private Cloud (GPC), a new virtual machine (VM) environment built using OpenStack technology. The GPC leverages the breadth of cloud functionality to serve a broader community with a more self-provisioning approach. To complete the continuum, the NCCS is evaluating commercial clouds, including cloud bursting to providers such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform.

Project Details

Discover, ADAPT, GPC, and commercial clouds are all successful in their own domains. The challenge is building interoperablity to enable migrating workloads to the optimal platform. Among the possible solutions are using container technology to house systems and offering a “single pane of glass” portal to all systems. The NCCS also seeks to demonstrate that HPC work can be done in commercial clouds, which historically has not been viable given the deployed interconnect technologies. However, the current trend shows cloud providers standing up high-speed, low-latency networks that can service MPI jobs. Extensive benchmarking of the different offerings will help characterize where to best run applications.

Results and Impact

When deployed, a seamless compute environment offering a mix of compute types—HPC, cloud, etc.—has the potential of significantly increasing the efficiency of NASA scientists. While efficiency may not always equate to lowest cost, time-to-science may be more important than minimizing dollars spent. The ability to select, configure, and customize the target compute environment will put unprecedented power in users’ hands.

Why HPC Matters

The combination of the Discover, ADAPT, and GPC on-premises clouds augmented by commercial clouds presents a diversified capability to users. When full integration is achieved, science users and the NASA missions they support can benefit beyond what was previously imagined.

More Information

Hoot Thompson, NASA Goddard Space Flight Center
hoot.thompson@nasa.gov