Science in Amazon Web Services (AWS): From JupyterHub to Clusters
Overview
NASA Goddard’s Science Managed Cloud Environment (SMCE), a project of the Computational and Information Sciences and Technology Office (CISTO), is a managed Amazon Web Services (AWS)-based infrastructure that leverages cloud computing capabilities for NASA-funded projects. The primary purpose of the SMCE project is to reduce the waiting time for NASA scientists and collaborators to gain access to the infrastructure and software that they need to conduct research. The SMCE’s AWS environment is designed to provide same-day cloud access to NASA principal investigators and their non-NASA team members. Cloud computing allows CISTO and other organizations to deploy infrastructure on demand and in record time. For example, new high-performance computing (HPC) clusters or JupyterHub ecosystems can be deployed by scientists in less than an hour.
Project Details
One key goal of the SMCE is to enable collaboration between NASA scientists and non-NASA researchers. Another SMCE goal is to minimize the time that the scientists and collaborators must wait to access their deployed infrastructure. By enabling virtual collaboration and providing faster access to HPC in the cloud, the SMCE meets both goals and increases the pace of research and discovery.
SMCE users have been able to access SMCE services within the same day of requesting those resources. Whether users need a highly available Git repository and Docker Registry, the ability to launch MPI clusters, or access to JupyterHub/Dask/Pangeo infrastructure, thanks to our focus on Infrastructure as Code (IaC) for scientific software, the SMCE can provide those resources in record time.
"The SMCE’s AWS environment is designed to provide same-day cloud access to NASA principal investigators and their non-NASA team members. Cloud computing allows CISTO and other organizations to deploy infrastructure on demand and in record time."
Understanding that scientists have different approaches and preferences, the SMCE team is also flexible in the way that we enable scientists to access this infrastructure, ranging from point-and-click interfaces to using Python code to deploy Dask clusters. Despite the agility of approaches and faster access, the SMCE does not sacrifice security for speed: we implement the same level of security required for any HPC production system. We also automate governance to make security and policies as transparent as possible for the end user. In the SMCE environment, every piece of infrastructure and application can be understood by looking at a Git (version-control system) repository.
Results and Impact
Due to the automation of governance and the use of IaC, the SMCE was able to deploy five JupyterHub infrastructures in record time (a matter of hours) for the five project teams of the virtual, global Heliophysics Hackweek 2020 (HW20) event hosted by the NASA Center for Climate Simulation (NCCS) in August. The SMCE provided same-day access to each project’s unique computational environments, and after the event was over and those resources were no longer needed, the resources were simply removed, saving costs. A similar use case occurs frequently during NCCS-provided Python training for NASA users: the SMCE team creates JupyterHub infrastructure in record time and decommissions it after training is completed, again optimizing cost.
Why HPC Matters
Having access to an HPC infrastructure provides NASA-funded scientists with a needed platform for speeding up research. The SMCE complements the NCCS Discover supercomputer and ADAPT Science Cloud by providing short-wait-time access to cloud-based HPC, making it ideal for projects where NASA scientists need to collaborate with scientists from other institutions.
Related Links
Ramon Ramirez-Linan, NASA Goddard Space Flight Centerramon.e.ramirez-linan@nasa.gov