AWS JupyterHub and ParallelCluster Architecture and Performance

Overview

NASA’s Science Managed Cloud Environment (SMCE) is a managed Amazon Web Services (AWS)-based infrastructure for NASA-funded projects. SMCE engineers at NASA Goddard Space Flight Center were challenged to integrate numerous existing open-source projects that can easily be replicated both in the cloud and on premises in a complementary fashion. The SMCE team developed the NASA Earth Information System (EIS): a flexible, rapid response computing capability that leverages the versatility of the AWS cloud, including high-performance computing (HPC) services. With an Open Science objective, the SMCE team designed a platform that creates Infrastructure as Code (IaC) artifacts that are useful to NASA scientists and allows organizations outside of NASA to replicate this deployment.

NASA’s Science Managed Cloud Environment (SMCE), created by engineers at NASA's Goddard Space Flight Center, integrates numerous existing open-source projects that can easily be replicated both in the cloud and on premises. The SMCE team developed the NASA Earth Information System (EIS): a flexible, rapid response computing capability that leverages the versatility of the AWS cloud, including high-performance computing (HPC) services.

Project Details

The new SMCE platform features Jupyter Notebooks that integrate with Dask technology as a launch point for Parallel Python and HPC batch submittals. The new platform also facilitates instantiating AWS Lambdas directly from a notebook, which enables running code on a serverless infrastructure in response to events such as a new dataset being uploaded to an Amazon Simple Storage Service (S3) bucket. The notebook can access shared Elastic File System (EFS) storage, which is also available to the HPC cluster environment. Lastly, the platform provides connection to any server/service implementing Git for version control.

Turning to the HPC side, over the past few years, AWS introduced innovations including the Elastic Fabric Adapter (EFA) for high-speed interconnectivity between compute nodes and Amazon FSx for Lustre backed by S3 for high-performance file systems. Combined, these capabilities position AWS HPC clusters to rival on-premises HPC performance. Leveraging the Jupyter Notebook framework as the user interface reduces the barrier to cloud-based HPC resources. Scientists can quickly launch HPC models from Python notebooks and analyze/visualize results as they are created.

Results and Impact

SMCE engineers collaborated with the EIS Pilot Study’s fresh water, sea level rise, and fire projects to demonstrate the viability of a combined Jupyter Notebook and HPC architecture using the Estimating the Circulation and Climate of the Ocean (ECCO) model as an example.

To validate the assertion that AWS is competitive with on-premises systems, the SMCE team instantiated clusters with EFA support, then benchmarked several NASA-relevant HPC applications including the NASA Advanced Supercomputing (NAS) Parallel Benchmarks, the Goddard Earth Observing System (GEOS), and the Weather Research Forecasting (WRF) model. The team ran file system benchmarks such as IOR to test Lustre write/read performance. AWS versus on-premises comparisons are available upon request.

Why HPC Matters

HPC is vital to meeting the computational demands of NASA’s environmental research objectives. NASA has many disconnected data repositories and models, and fully integrating them requires a new approach. Due to participation by a broader community, this game-changing open framework will spawn rapid growth and benefit a wide range of research, commercial, and governmental objectives.

What's Next

Future work by the SMCE team will include tightening the integration of the Jupyter Notebook environment with AWS resources to provide a more seamless experience for NASA scientists and other users. Computationally, new testing will include other processor options such as ARM, along with GPU-based systems for both HPC and machine learning.

More Information

• NASA Science Managed Cloud Environment (SMCE)
• NASA Earth Information System
• Estimating the Circulation and Climate of the Ocean (ECCO) model

Researchers

• Ramon Ramirez-Liñán, NASA Goddard Space Flight Center
  
  Please enable Javascript to see email address.
• Hoot Thompson, NASA Goddard Space Flight Center
  
  Please enable Javascript to see email address.