Making the Cloud Do What NASA Scientists Want

Overview

The NASA Science Managed Cloud Environment (SMCE) team has been working directly with multidisciplinary scientists to generate a cloud-native, collaborative environment for scientific research and development. The SMCE team has assembled JupyterHub, Python, Dask, high-performance computing (HPC) parallel clusters, and other commonly used tools into a rapidly deployable environment used for large-scale analytics, data processing, and artificial intelligence and machine learning (AI/ML) applications. This initial work lays the foundation for creating an open science analytics platform—the Elastic Science Cloud Analytics Platform (ESCAP)—deployable across NASA and non-NASA research institutions and fueled by Infrastructure as Code best practices.

Project Details

The collaboration between the SMCE team and NASA scientists has yielded numerous requirements and desired features that have guided the design and implementation of ESCAP. Through a close integration between JupyterHub and Parallel Cluster, users can submit and monitor HPC jobs while analyzing data as soon as it is created. The SMCE team is seeking to achieve an even closer integration between Jupyter notebooks and HPC resources through shared user and group identities and JupyterLab extensions to enable browsing and downloading both public (as offered by Amazon Web Services, Microsoft Azure, and Google Cloud) and private datasets accessible from the notebooks. Technologies being explored for integration include Eclipse Che and easier methods of introducing user-defined Dask containers. Ever in the picture is a secure environment that promotes user flexibility, collaboration, and rapid research and development, while minimizing risk.

Results and Impact

The outcome of this initiative will be a fully open-source, cloud-based solution that is easily and rapidly deployable and can be dynamically provisioned and shut down, as needed, to curtail unnecessary costs. This would create a reference cloud-based architecture for commercial cloud computing and on-premises computing capabilities for NASA scientists and collaborators. In addition, scientists could publish analyses, tools, and even web endpoints side-by-side with the data for users to quickly enable a wide range of use cases.

Why HPC Matters

The end game of the SMCE is to enable and promote reduced time to science. HPC is vital to meeting the computational demands of NASA’s environmental research objectives. Yet, since NASA has several disconnected data repositories and models, fully integrating them requires a new approach. Due to participation by a broader science community, this game-changing open framework will spawn rapid growth and benefit a wide range of research efforts.

What's Next

Future work by the SMCE team will include easing access to NASA’s vast data repositories for NASA scientists and other affiliated researchers and promoting a seamless processing experience. For single sign-on, the team will focus on facilitating data transit between computing platforms (HPC, Dask, Notebooks, etc.) by integrating authentication and authorization across all platform components. Computationally, new testing will include additional processor options such as ARM- and AMD-based architectures, along with GPU systems for both HPC and AI/ML.

More Information

• Science Managed Cloud Environment (SMCE) System Overview
• Earth Information System
• The Open Science Project

Researchers

• Hoot Thompson, NASA Goddard Space Flight Center
  Please enable Javascript to see email address.
• Daniel Duffy, NASA Goddard Space Flight Center
  Please enable Javascript to see email address.
• Ramón Ramírez-Liñán, NASA Goddard Space Flight Center
  Please enable Javascript to see email address.