Bursting into Public Clouds for High-Performance Analytics

Overview

The NASA Center for Climate Simulation (NCCS) is working to leverage the power of onsite resources coupled with cloud-based assets to expand—on a dynamic basis—the computational power afforded its user population. The objective of this "cloud bursting" is to demonstrate automated overflow processing from the NCCS Advanced Data Analytics Platform (ADAPT) to Amazon Web Services (AWS) using the SubmitOnce component of Cycle Computing's CycleCloud provisioning tool.

Project Details

The test case centers on using high-resolution satellite imagery and custom processing algorithms to identity and count trees and shrubs within a coast-to-coast swath of Africa from the Sahara south to the savanna zones. This biomass estimate will reveal how much carbon is stored in the region's vegetation that could be released as carbon dioxide if those plants burn or die and decompose due to natural or human causes and thus have direct impact on the region's climate.

Satellite imagery is organized into 11 Universal Transverse Mercator (UTM) zones, which are roughly 658 by 1,334 kilometers (km) each. UTMs are further broken down into 100- by 100-km tiles. Seamless sub-tiles with image distortions removed are created on ADAPT and then transferred to AWS for the biomass calculations. The batch queueing system in ADAPT communicates with its elastic twin in AWS to move data and initiate processing. AWS then returns results to an ADAPT file system.

Results and Impact

Satellite imagery is organized into 11 Universal Transverse Mercator (UTM) zones, which are roughly 658 by 1,334 kilometers (km) each. UTMs are further broken down into 100- by 100-km tiles. Seamless sub-tiles with image distortions removed are created on ADAPT and then transferred to AWS for the biomass calculations. The batch queueing system in ADAPT communicates with its elastic twin in AWS to move data and initiate processing. AWS then returns results to an ADAPT file system.

• Significantly increasing compute resources on an "as needed" basis.
• Decreasing the "time to science," making more iterations feasible.
• Managing cost by using AWS spot pricing.

Why HPC Matters

The combination of local and cloud resources provides scientists with an unparalleled tool that can scale as their processing demands evolve and grow. The ability to iterate and refine will yield more robust findings, greatly enhancing the underlying science.

What's Next

The next step is to move to a more dynamic or load-driven model of allocating jobs to ADAPT versus AWS. To date, jobs have split by type between ADAPT and AWS. In the future, jobs will automatically launch wherever resources such as CPUs and memory are most available and affordable.