NCCS—On the Open Road to OpenStack
Overview
Who would set themselves up to compete with three of the biggest, most successful companies on the planet? The NASA Center for Climate Simulation (NCCS) at NASA Goddard Space Flight Center did just that; we built an on-premises cloud environment to support collaborative science for those using NASA datasets. We started small, added a lot of NASA data, made a few friends (and mistakes!) along the way, and have outgrown our humble beginnings. When we liked what we saw, we set about building a new, open-source OpenStack cloud environment.
Project Details
The NCCS currently supports two OpenStack cloud environments: a Platform as a Service (PaaS) focused on science along with an Infrastructure as a Service (IaaS) focused on engineering. We are merging these two systems into one—called “Explore”—with over 9,000 cores providing 322 teraflops to our cloud users.
Explore uses a streamlined deployment method incorporating configuration fixes that have accumulated over the years. NCCS is in the process of upgrading our aging storage system to better support dynamically creating and destroying virtual machines (VMs), increasing security by using tenants with network isolation, and providing the high performance levels needed by artificial intelligence/machine learning (AI/ML) workflows.
The NCCS currently supports two OpenStack cloud environments: a Platform as a Service (PaaS) focused on science along with an Infrastructure as a Service (IaaS) focused on engineering. We are merging these two systems into one—called “Explore”—with over 9,000 cores providing 322 teraflops to our cloud users.
Results and Impact
Explore will be operational in late fall 2021, but administrators are already reaping the benefits of the new system. Specific time savers include using Ansible to manage the OpenStack Control Plane and Hypervisors, integration with NASA identity management tools, and faster VM creation through a modernized configuration management (CM) infrastructure. System administrators will also have the ability to leverage existing NCCS support services that were previously managed locally such as LDAP, logging, monitoring, and security scanning.
NCCS users come from diverse fields including climate modeling, Earth science, astronomy, and engineering. Our goal is to provide a seamless environment for these researchers’ ideas to grow into projects that leverage cloud computing, high-performance computing (HPC), shared NASA data products, and public data services. Those who migrate to Explore will be better able to manage their compute resources, with more options for quickly increasing or decreasing their assigned resources; fewer downtimes associated with storage issues; and a streamlined approach to accessing JupyterHub.
Why HPC Matters
HPC accelerates science, but the term “HPC” has become closely associated with supercomputers, MPI jobs, and batch schedulers—and thus a certain kind of workflow. However, many modern and emerging workflows are not necessarily a best fit for a supercomputer architecture. Cloud’s flexibility lends itself well to meeting these new patterns. The cloud is not competing with HPC. Rather, the cloud itself provides a kind of “HPC” that differs from that of a traditional supercomputer. The science institutions of the future will need to incorporate HPC and cloud in order to thrive, and OpenStack provides a framework for deploying both methodologies.
What's Next
After Explore goes into operations in fall 2021, we plan to take the lessons learned using OpenStack for Explore to identify technologies and processes that will benefit future system builds, monitoring, and maintenance in the NCCS Discover supercomputer. Beyond that, we plan to build an orchestration layer that allows users to access on-premises cloud and HPC via tools such as JupyterHub and to easily burst into public clouds, when appropriate.
More Information
Researchers
- Laura Carriere, NASA Goddard Space Flight Center Please enable Javascript to see email address.
- Robert Budden, NASA Goddard Space Flight Center Please enable Javascript to see email address.
- Jonathan Mills, NASA Goddard Space Flight Center Please enable Javascript to see email address.