Using NASA High-Performance Computers to Analyze Large Earth Science Data

Overview

The NASA Center for Climate Simulation (NCCS) is collecting major NASA Earth datasets in one location to help scientists conduct big data analytics using data from multiple NASA projects. Scientists, both internal and external to NASA, can access petabytes of NASA data through the Earth Data Analytics Service (EDAS) without having to download the data, or even write the analysis tools. EDAS enables them to run simple calculations such as minimum, maximum, and average, as well as string the calculations together in workflows to perform more complicated calculations such as anomaly and standard deviation. The version being released in December 2018 will include the ability to run machine learning operations.

Project Details

EDAS facilitates constructing high-performance parallel workflows by combining canonical analytic operations to enable processing big datasets within limited memory spaces and with interactive response times. Users access EDAS services via a Web Processing Service (WPS) Application Programming Interface (API) being developed in collaboration with the Earth System Grid Federation (ESGF) Compute Working Team to support ESGF server-side analytics. EDAS operations are built using the Python Dask and Xarray ecosystem to facilitate collaboration with the Earth science research community. Client packages in Python, Java/Scala, or JavaScript contain everything needed to build and submit EDAS requests by both vetted and unvetted users.

Results and Impact

In addition to the canonical operations, EDAS services include configurable, high-performance neural network learning modules designed to operate on EDAS workflow products. As a science technology driver, we have explored EDAS capabilities for forecasting the interannual variation of important regional-scale seasonal cycles such as summer monsoons. We trained neural networks to forecast All-India Summer Monsoon Rainfall (AISMR) one year in advance using (as input) the top 8–64 Principal Components of the global surface temperature and 200 hectopascal (hPa) geopotential height fields from NASA’s Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2) and NOAA’s Twentieth Century Reanalysis. The promising results from these investigations illustrate the power of easily accessible machine learning services coupled to huge repositories of Earth science data. The EDAS architecture brings together the tools, data storage, and high-performance computing (HPC) required for timely analysis of large-scale datasets, where the data resides, to ultimately produce societal benefits.

Why HPC Matters

NASA HPC provides EDAS access to a 9-petabyte filesystem being populated with collections of climate predictions and reanalyses, as well as Landsat and other very high-resolution imagery. This repository will continue to grow with the addition of weather forecast data in the near future.

What's Next

Combining EDAS with the NCCS Centralized Storage project, now in development, will provide access to over 15 petabytes of Earth data for analysis by both vetted and unvetted users. The NCCS development team will be adding other analytic and machine learning operations to EDAS. Using open source methodologies will facilitate collaborations between NASA and global climate scientists. Ultimately, the NCCS intends to build a community providing opportunities for discovery to scientists worldwide.

More Information

• See the poster for this demonstration (PDF, 1.8MB). As shown in the NASA booth at SC18.