Climate Analytics-as-a-Service: Making Climate Science Data Accessible

Overview

Earth system models generate datasets currently ranging in size from hundreds of terabytes to a few petabytes; in the age of exascale, these datasets will increase to tens of petabytes and more. Traditional methods of data analysis, where data are moved via the Internet to a scientist's local computing platform, will simply not be feasible. In this project, we couple a high-performance data analytics platform to an innovative set of algorithms to create Climate Analytics-as-a-Service. Using our web service application programming interfaces (APIs) or Python library, we have demonstrated that a user can complete the most common types of analyses with a speed-up of several orders of magnitude.

Project Details

The Advanced Data Analytics Platform (ADAPT) combines purpose-built virtual machines (VMs) on high-performance computing (HPC) nodes, 5 petabytes of data, and a high-performance Hadoop cluster. Users can adapt the VMs to match their workstations, eliminating the high barriers to using traditional HPC systems. The Hadoop cluster comprises 36 nodes (576 Intel Sandy Bridge cores), 2.304 terabytes of memory, and 1.2 petabytes of storage.

Climate data comprise a collection of variables that characterize the state of Earth's atmosphere at a given time (t) and place (x,y,z)-variables such as temperature, precipitation, wind direction and speed, relative humidity, atmospheric pressure, etc. We have implemented a set of simple operations that perform basic arithmetic and compute descriptive statistics (maximum, minimum, sum, count, average, variance, and difference) in parallel on the high-performance storage cluster. These operators take as input a variable name, a spatial extent, and temporal extent and return the requested value:

result ≤= operation (variable (t0,t1),((x0,y0,z0),(x1,y1,z1)))

Because of their widespread use, these simple operations function as "canonical operations" that can be used to build more complex expressions as well as workflows to generate trends, climatologies, and anomalies—among the most common types of data reductions used in climate science.

Results and Impact

Test cases have demonstrated that users can perform simple and complex analytical tasks significantly faster than traditional methods of data assembly, which typically involve iterative discovery, movement to local workstations, and client-side analysis. Examples include:

• A scientist reduced the data assembly time of their modeling and analytic work from approximately 10 days to 5 minutes.
• Another scientist completed an analysis project that was estimated to take 9 months in a mere 3 weeks.
• A project team was able to expand the scope of research by a factor of five in the same time frame.

Why HPC Matters

The results presented demonstrate the value of high-performance data analytics. It allows users to exploit the massive climate science datasets efficiently and easily with a simple, yet robust, set of canonical operations and eliminates data movement through the Internet.

What's Next

This work is moving forward on several fronts. These include expansion of the system's capabilities to include ensemble operations over multiple reanalysis datasets in the Reanalysis Ensemble Services project, the use of native NetCDF data in the Hadoop File system, and the development of other in-memory technologies such as Spark, which will enable near-real-time performance of our analytic services.