Data Analytics and Storage System (DASS)—Mixing POSIX and Hadoop Architectures

Overview

The resolution of NASA climate and weather simulations has grown dramatically over the past few years with the highest fidelity models reaching down to 1.5-kilometer (km) global resolution. With each doubling of resolution, the resulting datasets grow by a factor of eight in size. As climate and weather models push the envelope even further, a new infrastructure to store data and provide a platform for large-scale data analytics is necessary.

Project Details

The NASA Center for Climate Simulation (NCCS) has deployed the Data Analytics and Storage System (DASS), which combines scalable high-performance computing (HPC) storage with the ability to perform in-situ analytics. Within the DASS, large commonly used climate datasets are stored in a Portable Operating System Interface (POSIX) file system (write once/read many); examples of data stored within this system include data from the Landsat mission, the Modern-Era Retrospective analysis for Research and Applications, version 2 (MERRA-2) reanalysis, observing system simulation experiments, and high-resolution downscaled reanalysis data. The total size of this repository is on the order of 15 petabytes of storage.

In addition to the POSIX file system, the NCCS has deployed file system connectors to enable emerging analytics built on top of the Hadoop Distributed File System (HDFS) to run on the same storage servers within the DASS. Coupled with a custom spatiotemporal indexing approach, users can now run emerging analytical operations built on MapReduce and Spark on the same data files stored within the POSIX file system without having to make additional copies.

Results and Impact

Testing the Spatiotemporal Indexing Approach (SIA) under a variety of configurations, including HDFS, General Parallel File System (GPFS), and Lustre, has helped to clarify and define the architecture of the DASS hardware and the software stack. The DASS will provide engineers and scientists with a platform for analyzing large climate datasets without the need to move the data.

The transparency connectors allow system users to use both standard POSIX commands as well as perform Hadoop analytics on the same set of data. The ability to do this results in only needing one copy of the data.

Why HPC Matters

The DASS will provide the large capacity to store data for extended periods of time while also providing storage proximal computing capabilities to perform data analytics.