Exploring the Overlap of High-Performance Computing and Big Data for Climate

Overview

The NASA Center for Climate Simulation (NCCS) is constantly evaluating new and innovative technologies for potential use within its high-performance computing (HPC) environment. Traditionally, these technologies have focused on how to make models execute faster and scale to larger core counts. As NCCS has integrated new HPC capabilities, the data generated by the climate models running there has grown exponentially.

For example, the Global Modeling and Assimilation Office (GMAO) at NASA's Goddard Space Flight Center (GSFC) is currently generating a high-resolution, long-time-scale simulated climate. The GMAO will generate a 2-year Nature Run at a 7.5-kilometer (km) resolution and a 3-month Nature Run at a 3.5-km resolution. Seeded only by sea surface temperatures, these atmospheric simulations then run on their own to test how well the model reproduces past conditions. Approximately 4 petabytes of total data will be generated throughout these runs with the added requirement that the entire dataset be publicly available as soon as possible.

Project Details

This confluence of HPC and data service requirements has created a new context for evaluating technology. NCCS has developed a strategic view of information technology based on key architectural aspects of a mix between HPC and large Internet service providers.

By identifying key technologies of interest, these innovations can help fill in the gaps of HPC and data services and begin to evolve NCCS towards enabling new science. Key aspects of these technologies include using the data in place (i.e., moving the application to the data) and providing new methods to get to the data while enabling agile, flexible systems that can adapt to users' needs.

Results and Impact

NCCS has explored a number of different technologies to enhance its capabilities to provide both HPC and high-end data services for science. These include the creation of a prototype science cloud for use in satellite data processing and in support of engineering, remote visualization capabilities for climate, data management tools, the use of RESTful file systems for big data analytics, and the creation of a Climate Model Data Services Application Programming Interface (CDS API). Evaluation and testing of these technologies occurs within the NCCS architectural context. Current challenges include gaps between HPC and big data, which will be addressed by upcoming applied research.

Role of High-End Computing Resources

NCCS provides HPC resources to the NASA scientific community specifically designed to enhance our understanding of climate change and weather. Coupled with a robust and growing data analysis capability, NCCS provides an environment in which large-scale climate models are executed, data is analyzed and visualized, and discoveries are made.