Enabling AI/ML in an HPC Environment
Overview
For many decades, the world of high-performance computing (HPC) has relied on the use of Intel processors, high-speed interconnects, and large storage arrays to provide parallel computing to disciplines such as weather forecasting, climate modeling, N-body simulations, molecular biology, and the oil and gas industry. The last 5 to 10 years has brought substantial changes to this landscape, specifically the advent of artificial intelligence and machine learning (AI/ML) and its burgeoning application to science problems. This interest has fueled, and funded, an explosion of new GPU technologies that are particularly well suited to solving AI/ML problems. As we begin to see AI/ML find a serious role in science applications, we anticipate the need to combine traditional HPC and big data access with GPU resources configured with AI/ML tools. At the NASA Center for Climate Simulation (NCCS), we view the new demand for AI/ML as an opportunity for growth much like a chef branching into fusion cooking, with the whole being greater than the sum of the parts.
"At the NASA Center for Climate Simulation (NCCS), we view the increasing demand for artificial intelligence and machine learning (AI/ML) in science applications as an opportunity for growth. We are addressing that need by creating a big data environment that allows researchers to move seamlessly between traditional high-performance computing and AI/ML without having to do any data movement."
Project Details
The NCCS has collected the ingredients needed to enable AI/ML in our HPC environment, including a Centralized Storage System (CSS) that provides access to NASA-curated datasets and, as appropriate, training datasets. CSS is available to our traditional CPU-based HPC system, Discover; our private cloud, ADAPT; and our AI/ML GPU Cluster. We have ingested 12 petabytes of data into CSS and are providing software, where feasible, to generate analysis-ready datasets to further minimize the “time to science.” The NCCS brought the AI/ML GPU Cluster online in summer 2020. The cluster entails 22 nodes each containing 40 Intel cores, 4 NVIDIA V100 Tensor Cores, and 768 gigabytes (GB) of RAM. An additional 12 GPU nodes will arrive in winter 2021, comprised of 2 x 24 AMD Epyc Rome cores, 4 NVIDIA A100 cores, and 512 GB of RAM. These nodes will be configured as part of the next NCCS Discover Scalable Compute Unit expansion.
The goal of these NCCS acquisitions is to build a combined HPC, AI/ML, and big data environment that allows users to move seamlessly between traditional HPC and AI/ML without having to do any data movement.
Results and Impact
Among other applications, the addition of these capabilities has allowed researchers to use AI/ML for analyzing Transiting Exoplanet Survey Satellite (TESS) data to build nearly 70 million light curves using Discover’s CPU resources. From these light curves, the researchers have identified numerous potential planets, heartbeat stars, binary, triple, quadruple, and a sextuple star system (see Brian Powell’s SC20 abstract “Exploring the Stars with HPC”). A separate project used GPU resources to accelerate air pollution forecasts. The NCCS is branching out to support a new set of science users who will be leveraging tools such as TensorFlow, PyTorch, RAPIDS, EarthML, and scikit-learn to explore new research paradigms.
Why HPC Matters
Utilizing 74 CPU-years of traditional HPC with CPUs to generate the required TESS light curves was critical to the astronomy project. Having access to traditional HPC is also required for today’s climate models. Integrating these workflows with GPU workflows in one location means scientists can focus on the science rather than the compute.
What's Next
The NCCS plans to add more CSS storage and with it access to more data for users. We will also configure a test NVIDIA DGX system in fall 2020 and add GPUs to the Discover system in winter 2021.
More Information
Laura Carriere, NASA Goddard Space Flight Centerlaura.carriere@nasa.gov