Next-Generation Climate Data Services: MERRA Analytics

Overview

Scientific data services are critical to the mission of the NASA Center for Climate Simulation (NCCS). NASA's Modern Era Retrospective-Analysis for Research and Applications/Analytic Services (MERRA/AS) is a cyberinfrastructure resource for developing and evaluating the next generation of climate data analysis capabilities. MERRA/AS supports Obs4MIPs (observations for model intercomparison projects) activities by reducing the time spent in preparing MERRA data used in data-model intercomparison. It also provides a testbed for experimental development of high-performance analytics.

Project Details

MERRA/AS is a cloud-based service built around the Virtual Climate Data Server (vCDS) technology that NCCS currently uses to deliver Intergovernmental Panel on Climate Change (IPCC) data to the Earth System Grid Federation (ESGF). To maximize effectiveness, the MERRA/AS servers use a workflow-generated realizable object capability to perform analyses of the MERRA data using the Hadoop Map/Reduce approach to parallel, storage-based computation.

Results and Impact

The results produced by these capabilities will be stored by the vCDS, which will also be able to host code sets for scientists who wish to explore the use of Hadoop Map/Reduce for more advanced analytics. While the work described here will focus on the MERRA collection, these technologies can be used to publish other reanalysis, observational, and ancillary Obs4MIPs data to ESGF. Importantly, they offer an architectural approach to climate data services that can be generalized to applications and customers beyond the traditional climate research community.

Role of High-End Computing Resources

Powerful computing resources are necessary for on-demand analytic processing across 30-plus years of MERRA reanalysis data. The MapReduce operations leverage a cluster consisting of 36 Dell R710 servers, each populated with twelve 3-terabyte hard drives in addition to their operating system disks (CentOS 6.3). For interconnectivity, there is a 36-port Fourteen Data Rate InfiniBand (IB) switch plus a 48-port Gigabit Ethernet switch. Initial cluster metrics show one-way IB performance rates topping 6,000 megabytes per second, peaks of 314 gigaflops per server, and an overall capability of approximately 11 teraflops.