Supporting Petabyte-Scale Science on the NASA Earth Exchange

Overview

The NASA Earth Exchange (NEX) is a collaboration platform for Earth science researchers that combines state-of-the-art supercomputing, Earth system modeling, workflow and knowledge management, and NASA remote sensing data feeds. NEX provides a complete work environment in which users can explore and analyze large datasets, run modeling codes, collaborate on new or existing projects, and share results among the Earth science communities.

NEX supports over 20 large-scale science projects ranging from satellite data processing and analysis to machine learning. These projects produce volumes of data up to 25 petabytes (PB), and together consumed over 9 million computer processor-hours in the past year.

Project Details

The main goal of NEX is to enhance and enable more efficient use of Earth observations for NASA Earth science technology, research, and applications. NEX utilizes the Pleiades supercomputing platform, together with over 2 PB of satellite, climate, and model datasets, to accelerate research in the Earth sciences. The results and project details can then be shared with the community through the NEX portal, creating a web-based knowledge network.

NEX supports diverse Earth science communities in the areas of technology development, large-scale data processing systems, climate downscaling, data mining and data analytics, remote sensing, and ecosystem modeling, as well as applications in agriculture management. In order to continuously engage the community, we have developed OpenNEX, an outreach platform that provides access to video lectures, tutorials, hands-on labs, and sample applications that anyone can use to access and explore NASA datasets. OpenNEX is structured like a series of workshops, where each focuses on a particular theme.

Results and Impact

NEX strives to lower the barrier of entry to data- and compute-intensive science, and enable research teams to execute projects at ever-larger scales that would otherwise be unfeasible. Example of these projects include:

• Creating a tree-cover map across the United States using 1-meter resolution data and advanced machine-learning algorithms.
• Developing a set of high-resolution, bias-corrected climate change projections that can be used to evaluate climate change impacts on ecosystems, hydrology, and human health.
• Implementing a global satellite data processing and visualization pipeline for processing almost 25 PB of data.
• Developing a new high-resolution global vegetation biomass map by blending a variety of satellite datasets useful for global carbon monitoring.

Why HPC Matters

The Pleiades supercomputer at the NASA Advanced Supercomputing facility, combined with facility's massive data storage resources and high-speed network, enables the NEX team to engage scientific communities and provide them with the capability to execute modeling and data analysis on a grand scale that was not previously achievable by most scientists. NEX projects processed and analyzed over 3 PB of satellite data over the last year, using over 9 million processor-hours on Pleiades.