MERRA Analytic Services: Orchestrating Big Data with Climate Analytics-as-a-Service

Overview

Faced with unprecedented growth in the Big Data domain of climate science, we have developed Climate Analytics-as-a-Service (CAaaS). We focus on analytics because the knowledge gained from our interactions with Big Data ultimately produces societal benefits. We focus on CAaaS because we believe it provides a useful way of addressing the problem: a specialization of the business-process-as-a-service concept, which is an evolving extension of infrastructure, platform, and software services enabled by cloud computing.

Project Details

A virtual CAaaS is under development for NASA's Modern-Era Retrospective Analysis for Research and Applications (MERRA) data collection. The MERRA reanalysis integrates 30-plus years of observational data with numerical models to produce a global temporally and spatially consistent synthesis of 26 key climate variables. MERRA is of growing importance to climate change research and decision support applications.

MERRA Analytic Services (MERRA/AS) servers perform analyses over the MERRA data using the Hadoop/MapReduce approach to parallel storage-based computation. The virtual CAaaS technology stack can be deployed to the cloud or on local enterprise hardware. In order to knit these capabilities together and deliver them into practical use, we have built the Climate Model Data Services (CDS) application programming interface (API). This API supports web service access by consumer applications, a command line interface for basic CDS commands, and advanced programmatic access for Python-savvy users.

Results and Impact

MERRA/AS brings together several elements in a full, end-to-end demonstration of CAaaS capabilities: 1) high-performance, data proximal analytics; 2) scalable data management; 3) software appliance virtualization; 4) adaptive analytics; and 5) a domain-harmonized API. The effectiveness of MERRA/AS has been demonstrated in several applications. CAaaS lowers the barriers to and risks of organizational change, fosters innovation and experimentation, facilitates technology transfer, and provides the agility required to meet our customers' increasing and changing needs.

Role of High-End Computing Resources

Powerful computing resources are necessary for on-demand analytic processing across 160 terabytes of reanalysis data. The Hadoop/MapReduce operations leverage a computing cluster consisting of 36 Dell R710 servers, each populated with twelve 3-terabyte hard drives in addition to their operating system disks (CentOS 6.3). For interconnectivity, there is a 36-port Fourteen Data Rate InfiniBand (IB) switch plus a 48-port Gigabit Ethernet switch. Initial cluster metrics show one-way IB performance rates topping 6,000 megabytes per second and an overall capability of approximately 11 teraflops.